Saravana Kumar

Checking Live Writer after blog server crash

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Fri, 30 Oct 2009 10:33:20 +0000

Life is not easy being a part time web administrator!! :-)

BAM - Production environment management

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Thu, 08 Oct 2009 10:36:00 +0000

Reposting after server crash!!

Scenario: In this article we are not going to discuss about how to utilize BAM for your BizTalk applications. I've made the assumption; you created few BAM activities and utilized them inside your BizTalk applications and the system is now running on your productions servers. In my experience, majority of the clients don't take advantage of BAM, when they do they just stick to the basics of creating few BAM activities and utilizing them in their applications. So this article is focused only on this basis and we are going to ignore all the fancy BAM stuff like BAM Views, Alerts, Notifications, real-time data etc.

Production Configuration:

At the very high level your production BAM configuration should like the one shown below

Step 1: Make sure you deploy the BAM activities:

This one might look silly :-) but it's possible you might have developed your solution taking advantage of BAM and not deployed BAM activities in your production environment. BAM failures are not treated very seriously by the BizTalk runtime. For example: Your orchestration might have BAM code embedded inside, but you haven't deployed the corresponding BAM activities. In this case the orchestrations will complete successfully without getting suspended but there will be lot of events in the event viewer notifying the BAM failures and also records will start accumulate in the TDDS_FailedTrackingData table in the database. Over a period of time this table will grow substantially, resulting in an unhealthy database and its not nice to see lot of failure events in your event viewer, which will interfere with analysing the real issues.

Step 2: Make sure TDDS sub-service is running.

When you configure a BizTalk host with the option "Allow Host Tracking" to true and created a host instance for that host, then you have created a TDDS sub-service inside that host instance automatically.

The important step is to make sure there is at least one BizTalk Host/Host Instance configured to run "Tracking Data Decode Service (TDDS)" in your environment. When you utilize BAM inside your BizTalk applications (inside your Orchestrations, Pipelines etc) BizTalk runtime will utilize one of the asynchronous event stream to write BAM data for performance reasons. When an asynchronous event stream is used, the BAM data will be first written to the BizTalk message box database and its the job of the TDDS service to move the BAM data from BizTalk message box database to BAMPrimaryImport database. If you don't run the TDDS service the data will start accumulating in the message box database and over a period of time will result in a bloated unhealthy message box database. For more info about Tracking host read this article http://msdn.microsoft.com/en-us/library/ee308950(BTS.10).aspx

Step 3: Setup your active window for the BAM data:

Based on your business needs, it will be critical to keep the historical BAM data for certain period. The default setting by standard BizTalk installation is to keep the data for 6 months. If you are capturing lot of BAM data, you might need to reduce this value to days, weeks or months. You can change the value by using the bm.exe tool.

You can see the existing value by executing the following command

bm get-activitywindow ?Activity:your_activity_name

You can set the active window to desired value by executing the following command

bm set-activitywindow ? Activity:your_activity_name ?TimeLength:1 ?TimeUnit:Day

Step 4: Make sure BAM_DM_<<activityname>> SSIS packages are configured to run

One of the biggest advantages of BAM is it creates the database infrastructure for you to capture business data. You don't need to manually provision the database tables and views; it's automatically created when you deploy the BAM activities using the management tool bm.exe. For every activity deployed there will be 5 corresponding database tables created in the BAMPrimaryImport database with the postfix _Active, _ActiveRelationships, _Completed, _CompletedRelationships and _Continuations (few SQL views will created to work with these tables and corresponding BAM_DM_<<activity name>> SSIS package will be created).

The main reason for this provisioning is to improve the read/write speed while accessing the BAM data. The number of records in the _Active table will be less; it will contain only in-flight instance data, once the instance complete processing the data will be moved to _Completed tables.

This active/completed partition is good to certain extend, until the number of records in your completed tables started to grow, that's where BAM_DM_<<activity names>> SSIS packages comes into picture. This SSIS packages are mainly responsible for 2 tasks:

Taking the database table provisioning to the next level, in order to improve read/write access speed, and
Removing the old data from the BAMPrimaryImport database. It can either be moved to BAMArchive database or purged as shown in the picture (beginning of the article).

Whenever you run this SSIS package additional tables are created in BAMPrimaryImport database post fixed by a GUID and old data is removed based on the active window configuration.

The SSIS packages for your BAM activities are created when you deploy your activities, but they are NOT configured automatically in your environment. In your production environment it's critical to create a SQL job to run the SSIS packages in a periodic basis (every day or week based on your BAM data volume).

NOTE: If you have installed your BAM databases on a named SQL instance, then you won't see the BAM_DM_ packages automatically when you connect to Integration services. See the topic "Where are my BAM_DM_ <<BAM Activity Name>> packages?" topic in the end of the article for the solution.

Step 5: Whether to archive it or purge it?

Historical Data:

Historical Data in BAM falls under 2 categories.

Based on the active window settings the data will be kept in BAMPrimaryImport database for that period. Example: If your active window is set to 5 days, then 5 days BAM data will be held in various activity tables (partitioned with multiple GUID?s) within BAMPrimaryImport database.
Any data that's older than the active window will be held in the BAMArchive database until its cleared manually.

It will be a business decision to keep the historical data for governance purpose. Until recently, before Microsoft issued the hot fix 971984 (http://support.microsoft.com/kb/971984 ) it was not possible to purge the data directly from BAMPrimaryImport database in BizTalk Server 2006. You always need to move the old historical data to BAMArchive database and then purge it manually (see the script in the bottom of the article for doing it).

With the hot fix (its available by default on BizTalk 2009) you can configure whether or not the data goes into BAMArchive database by using the bm.exe tool.

bm set-archive ?Activity:your_activity_name ?shouldArchive:true|false

Where are my BAM_DM_ <<BAM Activity Name>> packages?

If you have installed BAM databases on a named SQL instance and try to connect to Integration Services and expand the MSDB node, you'll see the following error message.

The SQL server specified in SSIS service configuration is not present or is not available. This might occur when there is no default instance of SQL Server on the computer. For more information, see the topic "Configuring the Integration Services Service" in Server 2005 Books Online.

As noted in the error message the solution is available under Configuring the Integration Services article. You need to open the MsDtsSrvr.ini.xml file located under the folder %ProgramFiles%\Microsoft SQL Server\100\DTS\Binn (90 instead of 100 if you are using SQL 2005) and put the correct server name\instance name inside the <ServerName> element.

You need to restart the SQL Integration Service after making this change.

BAMArchive Purge Script:

There is no clear procedures documented in dealing with the data present in the BAMArchive database. Over a period of time BAMArchive database will also grow to substantial size and it will be required to truncate the data. Here is the script I use to truncate the BAMArchive database, please note this will wipe out all the data present in the BAMArchive database (use it at your own risk), but you will still have the historical data based on your active window configuration in the BAMPrimaryImport database.

truncate table bam_<<your_bam_activityname_1>>_Instances
truncate table bam_<<your_bam_activityname_1>>_Relationships

truncate table bam_<<your_bam_activityname_2>>_Instances
truncate table bam_<<your_bam_activityname_2>>_Relationships

<<repeate the truncate table steps for all of your bam activities>>

Declare @SqlQuery nvarchar(4000)
Declare @num int

Set @num = (Select Count([Name]) From sys.tables
Where [Name] like '%bam_%Instances_200%-%-%' Or
[Name] like '%bam_%Relationships_200%-%-%')

-- Print @Num
Declare @i int
Set @i = 0
While @Num > @i
Begin
    Set @i = @i + 1
    Set @SqlQuery = (Select Top 1 'Drop Table [' + [Name] + ']' From sys.tables
        Where [Name] like '%bam_%Instances_200%-%-%' Or
        [Name] like '%bam_%Relationships_200%-%-%')
    Execute(@SqlQuery)
End

Nandri!

Saravana

BizTalk Videos and Posters

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Thu, 01 Oct 2009 07:40:48 +0000

Microsoft web world is doing this for a very long time now, publishing short videos ("How Do I ?") that will help users to kick start the concept quickly rather than going through tons of articles. BizTalk has entered this arena recently (relatively!!). Here is the URL for "BizTalk How do I?" video http://msdn.microsoft.com/en-us/biztalk/dd849956.aspx

A picture is worth 1000 words, most of us know BizTalk team is been publishing posters that explains specific part of the product architecture in detail. There are lot of updates and new posters from the team. You can grab it from the following URL?s:

BizTalk Server 2009 Posters
BizTalk Server 2006 R2 Posters
BizTalk Server 2009 Posters on MSDN
BizTalk Server Posters on Microsoft.com

Don't forget, you can also get to the posters easily via biztalk247.com link http://www.biztalk247.com/posters.aspx

Nandri!

Saravana

Marriage between BizTalk Server 2006 and IBM WebSphere Message Broker

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Mon, 28 Sep 2009 22:54:00 +0000

Introduction:

In my current client, this is the situation: There are two separate teams involved in two different projects. One utilizing BizTalk Server 2006 building BPM solution and the other utilizing IBM WebSphere Message Broker building an even management system. The day has arrived, where we need to make these two systems talk to each other via MQ. Looking at very high level picture wearing an architecture hat this is what you would have drawn on a white board:

Easypeacy, BizTalk got an MQ adapter, so it can put messages into MQ and WMB will pickup the messages from there and both the system can talk to each other seamlessly. But it just turns out; it's not as simple as that. In order for WMB to pickup the messages that come into MQ, the messages must be PUBLISHED rather than point-to-point PUT and while publishing certain TOPICS should be added. WMB in turn will do all its routing to the subscribers based on the topics that's present in the message.

It's always tricky if you don't know the semantics of glossary terms used by different teams. After some research and with the help of the Utility tool RFHUTIL.exe provided by IBM to write messages into MQ, it turns out the difference between PUBLISH and PUT is just the addition of some extra MQ specific headers. The format of the header is called MQRFH2; one of the segments in the header will contain the PSC commands required for publishing. Relevant TOPICS as name value pair can be added as part of the PSC command.

All these things will make more sense if we see some pictures. I utilized RFHUTIL to place couple of messages into MQ. When you browse the messages in MQ using MQ explorer, it will look like this:

You can spot the difference between the messages, the PUBLSIHED message format is MQRFH2 and the message data now contains some additional RFH information in the beginning.

When you double click and open the message, the first one (regular PUT) will look like this

And the second one (PUBLISH) will look like the one displayed below with additional MQRFH2 properties containing PSC information.

Also, if you look at the data you can see the additional information at the beginning of the message as shown below:

At this stage at least you know, what you are working on, what's your target goal is and now you need to concentrate on how you are going to manage it from BizTalk Server 2006.

Understanding BizTalk Adapters for MQ:

When it comes to BizTalk and MQ the clean way to integrate both of them is by using an adapter. BizTalk server 2006 comes with 2 different set of adapters to talk to MQ. The default MQSeries adapter which gets installed as part of the normal BizTalk installation and the second one MQSC adapter that comes as part of the "BizTalk Adapters for Host System" package. The major difference between the two is the first one can only be used on MQ hosted on windows environment, whereas the second one is used to talk to MQ hosted on windows/non-windows environment. The MQSC adapter is also called as client based adapter because its dependant on IBM MQ's client API's. There are two different flavours of MQSC client adapters, the one which supports transaction ? paid license required and the other without transaction ?free no license required (guarantees at-least once delivery, but cannot guarantee duplicate message delivery). Switching between the two types of client adapters is easy via BizTalk Send Port Configuration.

You can learn more about them here: http://blogs.msdn.com/biztalkcpr/archive/2008/10/07/snap-shot-of-biztalk-2006-solutions-for-mqseries.aspx, and http://msdn.microsoft.com/en-us/library/aa772116(BTS.10).aspx

A bit of research was required to settle for the right adapter, in our case it?s definitely a client based MQSC adapter since MQ was hosted on a non-windows environment.

Bonus Findings!!:

1. You can just install the WebSphere MQ client API's by doing running the WebSphere MQ 7.0 custom setup and selecting Windows client tools as shown below:

2. Once you have settled for the MQSC client adapter, you need to figure out the supported MQ platforms. According to the official document for MQSC adapter (http://download.microsoft.com/download/c/2/2/c22737c1-707e-42a3-ae45-5df40973a0a7/BizTalk%202006%20R2%20Datasheet.pdf) that comes as part of BizTalk Server 2006 (Host integration server 2006) the supported platforms are 5.3 w/ Fix Pack 10 or higher, and 6.0 w/ Fix Pack 1 or higher. But during the prototyping, I figured out the adapter happily talks to MQ 7.0 as long as the MQ 7.0 client is installed. BizTalk 2009 with Host Integration Server 2009 officially supports MQ 7.0.

3. If in case you are working on the MQSeries server based adapter, you will experience lot of issues with strange error codes. Majority of them will be related to transactions and MSDTC. The easiest way to fix it during development and focus on the business problem is to disable transaction on the send and receive port. The other proper option is to configure MSDTC ?Security Configuration correctly as described here http://msdn.microsoft.com/en-us/library/aa544733.aspx

Setting up a development environment:

One of the biggest challenges in an integration project (like this!!) involving two different technology stacks is setting up the development environment. You will lose majority of the time sorting out the environment, rather than working on the real problem. In my case it was not any different and I settled for building a virtual machine with everything required. I installed the software in the following order.

1. Clean Windows XP SP3 Virtual machine (OS choice is yours)

2. Installed SQL Server 2005, and SP1 on top of it

3. Installed Visual Studio 2005 (including VC++ components), and SP1 on top of it

4. Installed BizTalk Server 2006 and configured it

5. Installed MQ 7.0 90 days trial version

6. Installed ?BizTalk Adapter for Host Systems?

7. Downloaded http://www-01.ibm.com/support/docview.wss?uid=swg24000637 IH03.zip file containing RFHUTIL.

Make sure BizTalk is able to communicate to MQ:

Once the environment is setup as explained in previous step, the next thing to do is to make sure BizTalk can communicate to MQ without any trouble. In order to test it, create a BizTalk application with a FILE receive port and MQSC send port and hook them together with appropriate filter condition (hint: BTS.ReceivePortName= ReceivePort1).

To setup MQSC send port, you need to configure 4 basic properties as shown in the below figure

Channel Name: This is the channel name of type server connection and its case sensitive, for testing purposes the easiest way to create a channel is just by enabling "Remote Administration" on the queue manager. Right-Click on the queue manager and select "Remote Administration", on the pop-up window click the button "Create" next to "SYSTEM.ADMIN.SVRCONN channel" and click ok.

NOTE: I don't think this is the safest or right way to create a server connection channel, but on a development environment it certainly works. You should contact your MQ administrator to setup a proper server connection channel.

Connection Name: It's in the format "server name(port number)". The default port when you create queue manager in MQ explorer is 1414.

You need to specify the queue manager name and queue name.

JMS MQRFH2 BizTalk Pipeline Component:

At this stage we have a working BizTalk application that's able to send messages to MQ via the MQSC adapter. Now raises the million dollar question, how are we going to add the required format and headers to message as discussed in the beginning of the article to PUBLISH the message into MQ. JMS MQRFH2 BizTalk Pipeline component, that comes as part of Microsoft ESB Guidance/Toolkit package comes to our rescue. It's worth understanding the significance of this pipeline component before we proceed further.

As part of Microsoft ESB Guidance 1.0 http://msdn.microsoft.com/en-us/library/bb931189.aspx, which was released after BizTalk Server 2006 R2 (later became Microsoft ESB Toolkit 2.0 http://www.microsoft.com/downloads/details.aspx?FamilyID=BC86CF1E-EF29-4B19-95F7-388F64555090&displaylang=en after the release of BizTalk Server 2009), Microsoft released JMS MQRFH2 BizTalk encoder/decoder pipeline component to help customers utilizing Java Messaging Service (JMS) for their integration needs. At this point it's worth understanding the basic structure of MQ message and how JMS integrates with MQ.

WebSphere MQ messages are composed of three components:

The WebSphere MQ Message Descriptor (MQMD)
One or more optional message headers (in our case MQRFH2 header)
The message body.

The message descriptor contains standard message properties, applicable to all WebSphere MQ messages. The second component, optional message headers (example MQRFH2, CICS etc) are application specific. For example when MQ is integrated with JMS, certain information are mapped into the MQRFH2 headers as shown in the below diagram.

The JMS MQRFH2 encoder and decoder BizTalk pipeline component helps customers to extend their JMS integration with Microsoft BizTalk server as shown in the below figure.

Using the encoder component the correct JMS message with relevant MQRFH2 header information can be constructed and submitted into WebSphere MQ, on the other hand BizTalk can receive JMS messages from WebSphere MQ and decode the JMS specific properties present in the MQRFH2 headers and write it to the context properties of the message.

Installing JMS MQRFH2 component for BizTalk Server 2006:

JMS MQRFH2 component was bundled along with ESB Guidance/Toolkit packages which were released and targeted for Microsoft BizTalk Server 2006 R2 and 2009.

One of the constraints I forgot to mention in the beginning was, our environment is still BizTalk Server 2006 (not a big surprise in big enterprise where product migration doesn't happen that often), which mean we are still on .NET Framework 2.0.

This constraint prevented me from installing the latest Microsoft ESB Toolkit 2.0; as soon as you start the installation it will halt straight away asking you to install .NET Framework 3.5. Also, the other important information, ESB Toolkit 2.0 is not an open source project anymore like its predecessor Microsoft ESB Guidance 1.0.

But still there is some good news; ESB Guidance 1.0 is still available for download with complete source code at http://msdn.microsoft.com/en-us/library/bb931189.aspx and the JMS MQRFH2 is a normal pipeline component without any dependency on either BizTalk 2006 R2 or 2009 specific functionalities. That means, you can take the code from the package, compile it and deploy it in your BizTalk 2006 environment.

Once you have installed the downloaded msi file, navigate to "C:\Program Files\Microsoft ESB Guidance 1.0 - November 2007" and unzip the file ESBSource. We are interested in only 3 BizTalk projects highlighted in the below picture.

ESB.JMS.Schemas.Property:

This project contains all the relevant context properties associated with MQRFH2 header. Open the project in Visual Studio 2005, open the project properties and change the application name to "JMS.ESB" (any name you wish). Compile and deploy the project.

ESB.JMS.PipelineComponents:

This is the core component, and compiling this project requires some attention. This project is developed as a C++ project due to its dependencies with IBM C header files to construct the MQRFH2 headers. If you have installed VC++ on your system, you shouldn't have any problem compiling it. As you probably know, any BizTalk Pipeline component assembly should be present in the GAC, but in order to put the assembly in GAC it needs to be signed with a strong name key. There are some constraints in signing the assembly generated by this project, which is documented here http://msdn.microsoft.com/en-us/library/ee250074(BTS.10).aspx

Once the project is compiled successfully, the easy option to add a strong name key to the compiled assembly is by issuing the following command

SN ?R "C:\Program Files\Microsoft BizTalk Server 2006\Pipeline Components\ Microsoft.Practices.ESB.JMS.PipelineComponents.dll" "C:\Program Files\Microsoft ESB Guidance 1.0 - November 2007\ESBSource\Keys\ Microsoft.Practices.ESB.snk"

The above command will re-sign the assembly with the new key file. You can then put the assembly into the GAC by using gacutils.

ESB.JMS.Pipelines:

This project contains all the support BizTalk pipelines utilizing the encode and decode pipeline components created by EMS.JMS.PipelineComponents assembly. This project is found under the samples\JMS\Source folder, open the project in Visual Studio 2005, open the project properties and change the application name to "JMS.ESB" (or whatever name you have selected while deploying ESB.JMS.Schemas.Property project). Compile and deploy the project.

The JMS MQRFH2 component logic:

The component first inspects the BizTalk context property MQMD_Format and if its equal to "MQHRF2 " (important:with 2 trailing spaces, and its NOT MQRFH2 but MQHRF2 due to header structure), then it applies the logic and adds the MQRFH2 headers else its just pass the incoming message as it is.

The component builds a default MQRFH2 Header using the IBM Header definition file "cmqc.h". Then the Message Context Properties are checked and for each of the MQRFH2_Properties:"MQRFH2_StrucLength" "MQRFH2_Encoding" "MQRFH2_CodedCharSetId? "MQRFH2_Flags" "MQRFH2_NameValueCCSID" "MQRFH2_Format" If the property is set, then the value is parsed to determine if it is the appropriate type, i.e. Integer or 6 Character String. If the value is a correct type, then the value is written to the MQRFH2 Header.

Once each of the fixed values is set, then the length of the "MQRFH2_NameValueData" variable is determined and written to the "MQRFH2.NameValueLength" value in the MQRFH2 structure. Finally the content of the "MQRFH2_NameValueData" is prepended to the outbound MQRFH2 structure.

This final structure is then pre-pended to the outbound message body.

The component also takes care of encoding rules during this process.

Steps to PUBLISH a message into WebSphere MQ?

As discussed in the beginning of the article, publishing a message into WebSphere MQ is nothing more than adding some additional MQRFH2 headers into the message. With the help of JMS MQRFH2 pipeline component the task is seamless. Here are the steps to successfully PUBLISH the message into WebSphere MQ

1. In your BizTalk Application add a reference to the BizTalk application "JMS.ESB" (refer to "Installing JMS MQRFH2 component for BizTalk Server 2006" topic), so that you get access to deployed schemas and pipelines.

2. Create a static one-way send port configured with MQSC adapter similar to the one discussed under the topic "Make sure BizTalk is able to communicate with MQ".

3. On the send port change the pipeline to JMS.PassThroughSend

4. Inside your Orchestration set the following context properties on the outgoing message. Build and deploy the orchestration

MSG_OUT(MQSeries.MQMD_Format) = "MQHRF2 ";

MSG_OUT(Microsoft.Practices.ESB.JMS.Schemas.Property.MQRFH2_NameValueData) = "<psc><Command>Publish</Command><Topic>Test</Topic></psc>";

NameValueData is where you specify the psc command with application specific topic information in a XML format.

NOTE: I assumed the readers are familiar with binding and running the BizTalk Application.

Run the application with a sample message and inspect the corresponding WebSphere queue using MQ explorer.

The queue will resemble to the one shown in below figure with all the relevant MQRFH2 properties.

Issues Encountered:

There were couple of issues which returned the error reason code 2142 and 2085. 2142 was mainly due to MQRC_HEADER_ERROR, By hard coding the CodedCharSetId value to 437 the issue was resolved. 2085 was mainly due to UNKNOWN OBJECT, which was resolved by hard coding the StructLength to 96 (based on the psc command length). Still some minor work required to remove these hard coded values, but it should not be a big show stopper.

Conclusion:

The article summarised how custom MQRFH2 headers can be added to the WebSphere MQ messages utilizing the JMS MQRFH2 pipeline component that comes as part of ESB Guidance 1.0 and ESB Toolkit 2.0. The article also explained how the component can be utilized in a BizTalk server 2006 environment by obtaining the source code. Even though the original component was written to transmit and receive JMS specific header in the MQ message, in this article we have seen how it can be utilized to PUBLISH messages into MQ.

The end result is, it would have been impossible to achieve it without the JMS MQRFH2 component that comes as part of ESB Guidance/Toolkit. This article is bit long, but I hope reading a document for couple of hours is better than spending 3 days finding a solution :-)

Nandri!

Saravana

Temporary Post Used For Theme Detection (c7f1455c-4e85-45e5-99ff-f43ca3426ac7 - 3bfe001a-32de-4114-a6b4-4005b770f6d7)

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Sun, 27 Sep 2009 08:58:24 +0000

This is a temporary post that was not deleted. Please delete this manually. (e98e906f-0c87-48d4-a9c9-fdaf2ed54c1b - 3bfe001a-32de-4114-a6b4-4005b770f6d7)

SOA Initiative: Get your schemas and service contracts correct.

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Wed, 29 Jul 2009 12:38:34 +0000

SOA in an organisation is a journey; an organisation can't change their existing strategy overnight. There are tons of articles written about SOA initiative, so I'm not going to add one more. In this article I'll assume, your organisation has decided to implement SOA and as part of the journey you decided to implement your first service. Your journey is not going to stop after building your first service, so the organisation should plan and impose all of their SOA strategies and design principles when building their first service and treat it as a base platform for the forth coming services.

When it comes to building services, the basic building blocks are schemas and service contracts. In technical terms it translates to xsd's and wsdl's. Often times refactoring schemas will have a ripple effect right up to the top level, so it worth putting enough time and effort in nailing down your schema (data) and service contract requirements.

We'll just assume this sample scenario: Your organisation is building an address service with a single operation called FindAddress. Using the service, consumers can search for addresses based on building number and post code (SearchRequest). The service in turn will return the list of addresses (SearchResponse) that matched the criteria. The scenario is simple, so we can concentrate on the core principles. The structure of our Address Service looks like this

First of all, let's break down the things required to construct our Address Services.

We need a unique namespace to uniquely identify our Address service.
We need unique identifier to access our operation FindAddress. This in web services world will be SOAPAction. Normally its good practice to club the service namespace with the operation name to generate this identifier.
We need a request (SearchRequest) and a response (SearchResponse) message structure (types) required for our operation FindAddress.

There are various things you need to consider while designing the schemas and service contracts. Some of them will look so obvious and basic; things like folder structure, file names, namespace, placement of complex data structures, taking advantage of object oriented support for XSD's etc. But they will prove very critical when the services start to evolve.

The basic building block required for our address services is the request (SearchRequest) and response (SearchResponse) message structures. It was not so uncommon people dived directly into tools like Visual Studio, open up a .asmx/WCF web project or similar tools/techniques in other platforms and start building (implementing) the service before finalising the contract. The tools in turn auto generate the service contracts (wsdl). It's a great help from the tooling point of view, but its not going to help us in long run. Maintainability and versioning will become a major issues, since the service contract and service implementation are tightly couples. A better approach will be to build the schemas and contracts in a modular way, so each module can evolve independently with various version numbers.

I just picked up this quote (while reading a post from Martin Fowler) from Kent Beck, which I feel is very relevant, while designing the schemas:

People can read your programs much more quickly and accurately if they can understand them in detail, then chunk those details into higher level structures. --Kent Beck

Basically Kent Beck is insisting on breaking large methods into well named smaller methods, a technique he refers to as Composed Method Patterns. That's exactly what we are going to do here, we are going to break a large schema structure into smaller more maintainable structures in various files.

The below figure explains what we are trying to achieve in a nutshell:

Figure 1

When we translate the above picture into technical terms using xsd, wsld, import, data types etc. It will look like the one shown below.

Figure 2

The hierarch and complexity of having multiple files to construct such simple message types might look overwhelming initially, but trust me it's going to pay off in long run, when you start leveraging more services in your organisation.

Target Namespaces for various levels

Filename	Target Namespace
AddressService_1.wsdl	http://yourorg.com/wsdl/addressService/1
AddressService_1.xsd	http://yourorg.com/schemas/service/address/1
Address_1_0.xsd	http://yourorg.com/schema/entity/address/1.0
OrgDataTypes_1_0.xsd	http://yourorg.com/schema/entity/common/1.0

The numbers at the end of the namespaces and file names are there to mainly help us in versioning schemas and service contracts. I'll explain them in my next article. For the time being please ignore them.

Organisation wide common data models (OrgDataTypes_1_0.xsd):

The very first thing you need to do before constructing any services in your organisation is to find all the common entities (data structures) that will be required across the organisation. It will be difficult to identify all of them at on go, but you can always start with the basic list and add more structures to it as and when you identify them. That's the whole point of having those version numbers at the end and in the target namespace.

Majority of the types in OrgDataTypes_1_0.xsd will have simple or complex type with restrictions appropriate for your organisation. Whenever you are constructing complex data models, don't be tempted to use the standard xsd types directly inside your complex types. Example: While constructing the UKAddress type which requires a postcode, you could have easily end up using xsd:string type for post code. But in our solution we defined a simple type called PostCode which is an extension of xsd:string with a regular expression pattern restricting to only UK post code. In this way your schema validation will be effective and be able to capture the errors much earlier in the cycle. Also it will be a consistent pattern across the organisation.

<xsd:simpleType name="Postcode">
<xsd:restriction base="xsd:string"> <xsd:pattern value="((([A-Z]{2}[0-9]{1,2})|([A-Z]{1,2}[0-9][A-Z])|([A-Z][0-9]{1,2}))\s([0-9][A-Z]{2})|(BFPO\s\d{1,4})|(GIR\s0AA))"/> </xsd:restriction>
</xsd:simpleType>

Another important reason to build the common types used across the organisation is reusability; these types may be required across many services you build in the future. Example: The PostCode type may be required in the InventoryService to determine the stock location post code.

It's important you decide on the format of the namespace (target) you want to use. For our example I've chosen http://yourorg.com/schema/entity/common/1.0

Address Service specific data models (Address_1_0.xsd):

After defining the common entity structures, the next step is to build the entity types required for our address service. Address_1_0.xsd file will contain only service specific simple and complex types required to construct the address service. The first thing you need to do is to import OrgDataTypes_1_0.xsd file into Address_1_0.xsd, which contains all the basic building blocks (entities) used common across the organisation.

<xsd:import namespace="http://yourorg.com/schema/entity/common/1.0" schemaLocation="../../Common/1.0/OrgDatatypes_1_0.xsd"/>

Once the common entity schema is imported, the next important thing to do is to decide on a unique namespace (target) for the service specific schema Address_1_0.xsd. Make sure it's different from the one we have used for the common entity schema. For our example I've chosen http://yourorg.com/schema/entity/address/1.0.

Now you can start defining the entities required for the address service inside Address_1_0.xsd. Example: As shown in figure 2, Address_1_0.xsd contains the complex type UKAddress to represent a UK address structure, which is basically constructed using all the common entity types found in OrgDataTypes_1_0.xsd.

Address Service specific messages (AddressService_1.xsd):

This is the place where we construct the types that will be used directly by the service contracts WSDL. Similar to what we have done while constructing Address_1_0.xsd, the first thing we'll do here is to import the required schemas (Address_1_0.xsd and if required you can download the OrgDataTypes_1_0.xsd as well if you are referring to types directly from common entities).

In AddressService_1.xsd we will define types that are explicitly required to build the service contract (ex: SearchRequest, SearchResponse). Majority of the times these types won't have any reusable value outside the context of the service. Types like service specific errors, warnings, etc are good candidate for this file. Again it's essential to choose a consistent and unique target namespace format. For our example I've chosen http://yourorg.com/schemas/service/address/1.

Constructing Service Interface (AddressService_1.wsdl)

At this stage we got all the required building blocks to construct our Address Service WSDL. You can hand craft it, if you are familiar with the semantics of WSDL. Otherwise you can use the tools like XmlSpy, ThinkTectures WSCF (web service contract first) etc to construct the service WSDL.

Folder Structure:

Its also important to place the schemas and wsdl files in the right hierarch in the files system and contract repositories. You'll referencing/importing schemas regularly in order to construct new service specific schemas and contracts (WSDL's). Ex:

<xsd:import namespace="http://yourorg.com/schema/entity/common/1.0" schemaLocation="../../Common/1.0/OrgDatatypes_1_0.xsd"/>

The below figure shows the simple structure utilized for this sample scenario.

Advantages of taking this modular approach:

There are various advantages in building the schemas and contracts in a modular way. Let see some of the key values

1. Reusability: The number one reason is the reusability of types. We have seen the organisation wide types (town, postcode etc) are being reused when constructing the UK Address type in address service. In the future some other service like example Inventory can reuse the UK Address type directly by importing the address schema (with appropriate version number).

2. Versioning: Individual modules can evolve independently with different version numbers. Providing greater degree of maintainability.

3. Extensibility: XSD supports object oriented approach while designing schemas, which can be utilized to extend the base types and create much more complex types if required.

Nandri!

Saravana

BizTalk 247 blogDoc Update #11

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Sun, 26 Jul 2009 20:00:00 +0000

Update #11 for http://blogdoc.biztalk247.com Contributors: 109, Articles Count: 994

Articles Updated:

Elton Stoneman

AIC 2009: A Value-Driven Approach to ESBs

BizTalk Server/Planning and Architecture/ESB

BizTalk Cache Adapter Samples

BizTalk Server/Development/Adapters/General, Custom

BizTalk Cache Adapter

BizTalk Server/Development/Adapters/General, Custom

Using WCF-Custom Bindings in Dynamic Ports

BizTalk Server/Development/Adapters/HTTP, SOAP, WCF

Using PowerShell in BizTalk Post-Processing Scripts

BizTalk Server/Development/Power Shell

Walkthrough: Self-Certifying SSL in an IIS Web farm

BizTalk Server/Security

get on the bus

Connecting SalesForce notifications WebServices to BizTalk Server 2006 ? Blog Entry

BizTalk Server/Third Party Vendors

Increasing BizTalk development productivity by removing DLLs from the GAC

BizTalk Server/Development/Tips

File System permission for the BizTalk Server 2004 and BizTalk Server 2006 file adapter ? Blog Entry

BizTalk Server/Development/Adapters/FILE, FTP

Handling Trailing Delimiters in HL7 Messages with BizTalk Server 2006 ? Blog Entry

BizTalk Server/Development/Accelerators

What I would like to see in the HL7 Accelerator 3.0!

BizTalk Server/Development/Accelerators

Checking for the existence of a property in a BizTalk Server 2006 Message ? Blog Entry

BizTalk Server/Development/Promoted Properties

The maligned Distinguished Field or why Distinguished Fields are cool in BizTalk Server 2004 and BizTalk Server 2006

BizTalk Server/Development/Promoted Properties

Storing information for BizTalk Server 2004 and BizTalk Server 2006 ? a new model is exposed ? Blog Entry

BizTalk Server/Development/Cross Referencing

Security Wizard - Understanding BizTalk Server 2006 Security ? Blog Entry

BizTalk Server/Security

Tallan

The blog of the BizTalk team at Tallan

Updategram with multiple tables with foreign keys

BizTalk Server/Development/Adapters/SQL

Setting up an HL7 Solution

BizTalk Server/Development/Accelerators

Deploy BizTalk policy using command line tool

BizTalk Server/Deployment

BizTalk 2006 and Active Directory

BizTalk Server/Administration

Debug BizTalk Orchestration in Production Server

BizTalk Server/Production

Bram Veldhoen

Recursive Orchestration

BizTalk Server/Development/Orchestrations

BizTalk ESB Guidance 2.0 - Fixing the Itinerary OnRamp?

BizTalk Server/Development/ESB Guidance

Messaging-only request-response correlation

BizTalk Server/Development/Messaging

Post-receive debatching without orchestrations

BizTalk Server/Development/Messaging

Unit testing BizTalk artifacts using BizUnit

BizTalk Server/Testing/BizUnit

Ben Simmonds

bEncode

MIME Decoder and 64bit

BizTalk Server/Development/Pipelines, Pipeline Components

ESB Guidance Package

BizTalk Server/Development/ESB Guidance

ESB Guidance ? Dynamic Resolution

BizTalk Server/Development/ESB Guidance

Orchestration Deployment + Bindings

BizTalk Server/Deployment

Sequential Receives

BizTalk Server/Troubeshooting/Orchestration

Mark Coleman

Biztalk Development

Default content in Maps

BizTalk Server/Development/Maps

Unit Testing With XLANGMessage

BizTalk Server/Testing

BizTalk 247 blogDoc Update #10

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Mon, 13 Jul 2009 22:14:00 +0000

Update #10 for http://blogdoc.biztalk247.com Contributors: 103, Articles Count: 962

Articles Updated:

BizTalk Core Engines WebLog

Whole buncha stuff

BizTalk Server/Development/Tips

BizTalk Adapter Development

Good as Gold! New WCF Adapter FAQ Whitepaper!

BizTalk Server/Development/Adapters/WCF LOB Adapter/Information Posts

New! Line of Business Adapter Sample Walkthroughs and Whitepapers

BizTalk Server/Development/Adapters/WCF LOB Adapter/Information Posts

New whitepaper posted! "Modifying Messages with the WCF Adapters"

BizTalk Server/Development/Adapters/WCF LOB Adapter/Information Posts

WCF Adapter Video: How a WCF Receive Adapter Processes an Incoming WCF Message

BizTalk Server/Development/Adapters/WCF LOB Adapter/Information Posts

WCF LOB Adapter SDK Download

BizTalk Server/Development/Adapters/WCF LOB Adapter/Information Posts

WCF LOB Adapter SDK and its relationship with WCF Channel Model

BizTalk Server/Development/Adapters/WCF LOB Adapter

WCF LOB Adapter Usage Patterns

BizTalk Server/Development/Adapters/WCF LOB Adapter

Arnulfo Wing

Biztalk Patterns

Distinguished fields of type xs:dateTime not Working on Orchestrations

BizTalk Server/Development/Promoted Properties

BizTalk Business Rules Engine Handy functions in .NET

BizTalk Server/Business Rules

Deploying Business Rules using C# in BizTalk 2006 R2

BizTalk Server/Business Rules

Importing BRE Vocabulary with Multiple versions

BizTalk Server/Business Rules

Error trying to re-install SQL Server 2005 on a Failover Cluster environment

BizTalk Server/Database

Rahul Garg

BizTalk Distilled

Simple example of using BizTalk Business Rules Engine (BRE) with Database

BizTalk Server/Business Rules

BizTalk 2009 supported upgrade paths

BizTalk Server/Migration and Upgrade

BAM_DM_ResendJournalActivity error on BizTalk 2009 Re-install?

BizTalk Server/Troubeshooting/BAM

BizTalk Server Performance

MaxDequeueThread and MaxReceiveInterval in the adm_Service table

BizTalk Server/Performance

Some Links To BizTalk 2006 Performance

BizTalk Server/Performance

Threads, DB sessions, and throttling

BizTalk Server/Performance

BizTalk 247 blogDoc Update #9

saravana.kumar.nospam@nospam.biztalk247.co.uk (saravana) — Sun, 12 Jul 2009 09:50:00 +0000

Update #9 for http://blogdoc.biztalk247.com Contributors: 101, Articles Count: 943

Articles Updated:

Mikael Hakanssons

Mikael H?kanssons Blog

What is a BizTalk Application?

BizTalk Server/Getting Started

Jan Eliasen

Just another BizTalk Guy

Looping around elements of a message, followup

BizTalk Server/Development/Tips

SQL Server Adapter Add Generated Item Wizard closes unexpectedly

BizTalk Server/Development/Adapters/SQL

Distinguished field not working

BizTalk Server/Development/Promoted Properties

Promoting reoccurring elements ? Part II

BizTalk Server/Development/Promoted Properties

Promoting reoccurring elements ? Part III

BizTalk Server/Development/Promoted Properties

Promoting values from destination schema of map on receive port

BizTalk Server/Development/Promoted Properties

Why use distinguished fields?

BizTalk Server/Development/Promoted Properties

Assigning the value of an xs:integer to an integer faling

BizTalk Server/Development/Maps

Combining two messages in one map

BizTalk Server/Development/Maps

Handling comma separated values inside a map, part I

BizTalk Server/Development/Maps

Solving the "If-Then-Else" problem in a map - part III

BizTalk Server/Development/Maps

Two nodes on same level with same name but different namespace

BizTalk Server/Development/Maps

Custom XSLT, Scripting functoid, or built-in functoids ? a question about religion?

BizTalk Server/Development/Functoids

Several custom scripting functoid with the same method

BizTalk Server/Development/Functoids

The Context Accessor functoid, part I

BizTalk Server/Development/Functoids

The Context Accessor functoid, part II

BizTalk Server/Development/Functoids

The Context Accessor functoid, part III

BizTalk Server/Development/Functoids

WCF LOB Adapter SDK and BizTalk Adapter Pack

Multiple ways to generate metadata / execute Stored Procedures in the SQL adapter

BizTalk Server/Development/Adapters/SQL

Multiple ways to Poll (Receive Location in BizTalk) in the SQL Adapter

BizTalk Server/Development/Adapters/SQL

Connect to WCF LOB Adapter hosted as WCF service using ClientCredentials

BizTalk Server/Development/Adapters/WCF LOB Adapter

Surfacing BizTalk Adapter Pack Adapters (WCF Bindings) as native BizTalk Adapters

BizTalk Server/Development/Adapters/WCF LOB Adapter

Things to consider when writing WCF LOB Adapters for consumption through BizTalk

BizTalk Server/Development/Adapters/WCF LOB Adapter

Throughput stalls when using adapters, developed against the WCF LOB Adapter SDK, in BizTalk

BizTalk Server/Development/Adapters/WCF LOB Adapter

Connection failure when using the WCF Siebel Adapter