Suleiman Shakhtour Integration Tips

BizTalk Server: Receiving SAP IDOC Messages

2015-10-08T02:34:00.003-07:00

Introduction

"Intermediate documents (IDOCS) are standardized EDI-like documents supported by SAP for asynchronously communicating with both SAP and non-SAP systems. IDOCS are used to send and receive business documents like sales orders to or from a trading partner’s SAP system or external program."

The below is a sample file of IDOC message

In this post, I will explain how to receive and process IDOC message using BizTalk.
Note that I supposed that you installed sap client and sap-wcf adapter in your development machine.

Problem

You have a new request that in your integration middle-ware that using BizTalk server needs to integrates with SAP ERP system which will generate outbound messages for external system using IDOC messages over file system or ftp..etc. As a role of BizTalk developer you need to know how to parse this message and process it into your internal business process.

Solution

BizTalk Server provides a nice wcf-sap adapter to connect to SAP and generate a meta data (schema) of related IDOC messages.

First, when you check the structure of IDOC message, you will notice that it is a flat file content

which means you need to have a flat file schema as shown in previous sample message image.

IDOC messages are complicated messages, so you don't need to build schema manually because it will take a lot of time to build such kind of schema instead you can use wcf-sap adapter to generate schema for you.

There are some tricks and tips you need to consider to generate a proper IDOC schema which I will explain in the followings steps:

Create a new BizTalk Project è right click project è select Add è Add Generated Item
In Add Generated Item Window select Consume Adapter Service è Consume Adapter Service
In Consume Adapter Service windows select sapBinding in Select a binding drop down list è click configure button
In Configure Adapter window security tab select User Name in Client credential type drop down list è enter User name and password as provided by SAP admin
Select URI Properties tab then set Application Server Host è Set System Number è Set Client è Set Connection type as provided by SAP team è Click Ok button. Note these are the minimum requirements you may need to fill other properties in case there are some special configuration in SAP server side
In Consume Adapter Service window click connect buttonè select Service (Inbound operations) from select contract type drop down list è Select IDOC node
Let's say that you interested in Address Master message which is is ARDMAS when you try to search on ADRMAS* you will find this
When you try to select message you will get the following error

Tip

Let's Say SAP team informed us to use ADRMAS03.V3(731) which is the latest version è select the message è Select Receive then click Add button then click ok button
Now you can see the generated files which contain 4 scehmas and Binding files.

Note: IDocOperation.ADRMAS03.731.3.Receive is the main schema that you want to deal with , IDoc.ADRMAS03.731.3 is schema used by previous schema, IDocSharedTypes which is datatype schema imported by IDoc.ADRMAS03.731.3 schema. You can ignore IDocOperation.ADRMAS03.731.3.ReceiveResponse schema and Binding file because we only receiving file over file system

When you check IDocOperation.ADRMAS03.731.3.Receive schema you will find out there is no a flat file tab and child delimiter 0xd 0xa which means it is expecting file that generated from Windows system which is Carriage Return Line feed (CR+LF), however in my case SAP in Linux os that generates file with 0x0A Line Feed (LF) delimiter inside the file content
We need to change Schema Editor Extensions to Flat File and we need to replace child delimiter 0xd 0xa with 0x0A in IDocOperation.ADRMAS03.731.3.Receive.xsd and IDoc.ADRMAS03.731.3.xsd , open files using xml editor then replace. The question what if you receive IDOC messages from different SAP providers one using window and other using Linux then you need to use a customer pipeline to replace new line delimiter instead of updating schema manually.
You can test your IDOC message by validating message using schema editor and setting Validate Instance Input type to Native and Input Instance Filename with path of your sample file
Now you need to create receive pipeline and add Flat file disassembler to Disassemble component and set Document Schema to IDocOperation.ADRMAS03.731.3.Receive schema data type
Now you are ready to receive idoc message from any File system, ftp, sftp ...etc , create related adapter and set your receive flat file pipeline then you can convert flat file file to idoc xml document

Conclusion

In this post I walked-throughed how to use wcf-sap adapter to generate IDOC schemas and I highlighted some tips and tricks for working IDOC message which I hope you will help you in any future works

BizTalk Server: Extending WCF Adapter

2015-08-05T04:11:00.001-07:00

Introduction

One of the good features of BizTalk server is the extensibility of WCF adapter. WCF (Windows Communication Foundation) enables developers to extend and updates its standard run time behavior with several extensibility points like behavior, binding, binding element, channels, etc.
In this post, I will explain the followings:

How to extend BizTalk WCF Adapter?
Illustrate how to overcome one of the limitations of WCF Adatpter

Problem

I faced an issue when I tried to integrate with a web service of one of our customers.
They sent us a web service url and this url is a long url (261 characters!!) when I tried to configure a BizTalk WCF adapter it gave me the following error as shown in figure 1:
"Error saving properties.(System.ArgumentException) Invalid address length; specified address length is 261 characters, limit is 256 characters."

Figure 1. Error Message

If you try to consume a web service using a console application you will not face any issue related to url length because there is not a limit for address property.However, BizTalk WCF adapter has a limit because BizTalk team defined End Point Address property of WCF Adapter as type of URI with length 256.

Solution

To work around this issue, WCF adapter provides the capabilities to be updated and extended.
What we need is to have custom endpoint behavior and to provide a dummy url to EndPoint Address URL and provide a real long address in an extended custom property

To prepare a custom endpoint behavior we need to followings the following steps:

Create a class library project and we will call it as TechnetWiki.CustomEndpointBehavior as shown in figure 2

Figure 2. Creating Class Library Project
Now we need to create 2 classes one to extend abstract class BehaviorExtensionElement to define custom property for our custom url and interface IEndpointBehavior to set a custom url to Endpoint url at run time.To use these abstract class and interface we need to add reference to System.ServiceModel dll as shown
Create a new class and we will call it as CustomBehaviorExtensionElement that extends BehaviorExtensionElement

using System;

using System.Configuration;

using System.ServiceModel.Configuration;

namespace TechnetWiki.CustomEndpointBehavior

{

    public class CustomBehaviorExtensionElement : BehaviorExtensionElement

    {

        private ConfigurationPropertyCollection properties;

        [ConfigurationProperty("CustomAddress")]

        public Uri CustomAddress

        {

            get

            {

                return (Uri)base["CustomAddress"];

            }

            set

            {

                base["CustomAddress"] = value;

            }

        }

        public override Type BehaviorType

        {

            get

            {

                return typeof(CustomEndpointBehavior);

            }

        }

        protected override ConfigurationPropertyCollection Properties

        {

            get

            {

                if (this.properties ==null)

                {

                    this.properties =new ConfigurationPropertyCollection

                    {

                        new ConfigurationProperty("CustomAddress",typeof(Uri),null,null,null, ConfigurationPropertyOptions.IsRequired)

                    };

                }

                return this.properties;

            }

        }

        protected override object CreateBehavior()

        {

            return new CustomEndpointBehavior(this.CustomAddress);

        }

    }

}
Create another class called CustomEndpointBehavior that extends IEndpointBehavior interface

using System;

using System.ServiceModel.Description;

namespace TechnetWiki.CustomEndpointBehavior

{

    public class CustomEndpointBehavior : IEndpointBehavior

    {

        public Uri CustomAddressUri { get;set; }

        public CustomEndpointBehavior(Uri customAddressUri)

        {

            if (customAddressUri == null)

            {

                throw new ArgumentNullException("customAddressUri");

            }

            this.CustomAddressUri = customAddressUri;

        }

        public void AddBindingParameters(ServiceEndpoint serviceEndpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)

        {

        }

        public void ApplyClientBehavior(ServiceEndpoint serviceEndpoint, System.ServiceModel.Dispatcher.ClientRuntime behavior)

        {

            serviceEndpoint.Address =new System.ServiceModel.EndpointAddress(this.CustomAddressUri);

        }

        public void ApplyDispatchBehavior(ServiceEndpoint serviceEndpoint, System.ServiceModel.Dispatcher.EndpointDispatcher endpointDispatcher)

        {

        }

        public void Validate(ServiceEndpoint serviceEndpoint)

        {

        }

    }

}
Sign assembly by defining a strong key for the class library project as shown in figure 3

Figure 3. Sign the assembly.
Add assembly to GAC either by adding assembly to resources of BizTalk Application in BizTalk administration as shown in figure 4 or by run gacutil -i "{AssemblyPath}" command.<

Figure 4. Adding Assembly to GAC
To register behavior extensions to WCF Adapter, you need to update machine.config in path "C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config" by adding the following configuration inside behaviorExtensions tag <add name="customEndpointBehavior" type="TechnetWiki.CustomEndpointBehavior.CustomBehaviorExtensionElement, TechnetWiki.CustomEndpointBehavior, Version=1.0.0.0, Culture=neutral, PublicKeyToken=415d904e2f9a9bcd" />
Note: in case your WCF-Custom handler runs under 64 bit you need to update the machine.config in the following path "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config"
Now we are ready to use extension in WCF adapter, you need to create WCF-Custom send adapter and set any dummy url in Address (URI) as shown in figure 5

Figure 5. Setting Dummy URL
Select Behavior tab and right click EndPointBehavior then select customEndPointBehavior which is the extension the we created and define in machine.config as shown in figure 6

Figure 6. Select Extention
Set the real URL which has the long url as shown in figure 7

Figure 7. Custom URL Setting

Conclusion

BizTalk Server enables developers to extend WCF adapter to overcome any limitations in WCF capabilities. In this post we explained one of the scenarios that needed to extend WCF to overcome the limitation of WCF Adapter and we walk-throw in the steps needed to register wcf extension in WCF-Custom Adapter.

Dynamic Mapping Resolver

2014-02-03T00:50:00.001-08:00

Introduction

Mapping is one of the main activities in any integration solutions. We always use mapping to transform an input message to another message format and structure. It is very crucial to have a dynamic solution as much as we can to let our solution easy to update and deployment process smooth and not affecting the existing code.

Where do we use mapping?

We use mapping in inbound maps in receive port as shown in figure 1

Figure 1. Inbound Maps

We can use mapping in outbound maps in send port as shown in figure 2

Figure 2. Outbound Maps

We can use mapping in orchestration as shown in figure 3

Figure 3. Mapping in Orchestration

Problem

In orchestration we can see a big problem that if there is any a new map that we need to use in orchestration then we have to change orchestration which means to recompile and a redeploy our solution and that will affect the existing BizTalk solution process. On the other hand, in inbound and outbound maps if you imagine that we have dozens of maps, it will be headache for BizTalk administrator and developer to make sure that all maps are configured correctly and sometimes we need an operator user to update a BizTalk application configuration from BRE or a web portal because he is not convenient with BizTalk administration console and what if you have some cases that you want to map depending on context or content of that message in inbound and outbound maps without using orchestration.

Solution

We can have a solution by applying a dynamic mapping resolver.

What is dynamic map resolver?

It is a mechanism of dynamically(at run time) associate map name from existing repository like business rule engine or custom configuration database table to transformer based on an identifier contained in each source instance message.

How to implement dynamic map resolver?

We need to call transformation code from .NET code in custom pipeline component. I used ILSpy to reflect the code of TransformStream method from Microsoft.Practices.ESB.Itinerary.Services assembly. This method takes as parameters a Stream containing the message to transform, a String containing the fully qualified name of a map deployed within BizTalk, a Boolean for validating map source with message type passed and reference to message type. The method returns a Stream containing the transformed document.

I created a map helper that will be called from custom pipeline component to transform the stream of source to the expected destination stream.

01.using Microsoft.XLANGs.BaseTypes;

02.using Microsoft.XLANGs.RuntimeTypes;

03.using System;

04.using System.Globalization;

05.using System.IO;

06.using System.Xml.XPath;

07.

08.namespace TechNetWiki.DynamicMappingResolver.Library

09.{

10. public class


        MapHelper

11. {

12. public static


        Stream TransformStream(Stream stream,

string mapName, bool validate, ref string


        messageType)

13. {

14.

Type
        mapType = Type.GetType(mapName);

15. if (null == mapType)

16. throw new


        Exception(

string.Format("Map {0} is not exist.", mapName));

17.

18.

TransformMetaData
        metaData = TransformMetaData.For(mapType);

19.

20.

SchemaMetadata
        schema = metaData.SourceSchemas[0];

21.

String
        sourceMap = schema.SchemaName;

22.

23.

SchemaMetadata
        targetSchema = metaData.TargetSchemas[0];

24.

25. if (validate)

26. {

27. if (string

.Compare(messageType, sourceMap,

false, CultureInfo.CurrentCulture) != 0)

28. throw new


        Exception(

string.Format("Unable to execute map,'{0}'."

29.

"The
        source document type,'{1}'does not match the maps's "

+

30.

"target
        source document type,'{2}'"

, mapName,

31. messageType,

32. sourceMap));

33. }

34.

messageType
        = targetSchema.SchemaName;

35.

36.

XPathDocument
        doc =

new


        XPathDocument(stream);

37.

ITransform
        transform = metaData.Transform;

38.

Stream
        output =

new


        MemoryStream();

39.

transform.Transform(doc,
        metaData.ArgumentList, output,

null);

40. output.Flush();

41.

output.Seek(0,
        SeekOrigin.Begin);

42. return output;

43.

44. }

45.}

Note that you have to add <BizTalkInstallationFolder>\Microsoft.XLANGs.BaseTypes.dll and <BizTalkInstallationFolder>\Microsoft.XLANGs.RuntimeTypes.dll
to your class library project to build map helper code.

Then we need to create a custom pipeline component by creating class library project that has a reference to the created project that including MapHelper code and add reference to these dlls <BizTalkInstallationFolder>\Microsoft.BizTalk.Pipeline.dll , <BizTalkInstallationFolder>\Microsoft.XLANGs.BaseTypes.dll.

We need to use any configuration repository to retrieve map information at run time like Business Rule Engine (BRE ) or custom configuration table in database . In this article, I am using Business Rule Engine to retrieve map information.

BRE implementation

I will retrieve mapping map name depending on the MessageType which represents the source message type.However, you can use any context information to retrieve map name or any content information of the message. I created schema for this purpose containing MessageType and MapStrongName
as shown in figure 4.

Figure 4. BRE Schema

Then I created BRE rules using Business Rule Composer as shown in figure 5.

Figure 5. Sample BRE Rule

Note: that the message type is formed as Namespace#SchemaRoot and map strong name is formed as MapName, assembly strong name.
You can use vocabulary to ease life of operator user too.

Dynamic map resolver in Inbound Maps

We need to create a custom pipeline component and adding this component to pre-assemble part in the receive pipeline
so first we need to implement IComponent which contains Execute where we need to call TransformStream method as shown in the following code

01.public IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)

02.

03. string messageType = null;

04. string messageSourceType = pInMsg.Context.Read("MessageType"

"http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString();

05. string

mapName = GetMapStrongName(messageSourceType);

06.

System.IO.Stream
        originalStream = pInMsg.BodyPart.GetOriginalDataStream();

07.

pInMsg.BodyPart.Data
        = MapHelper.TransformStream(originalStream, mapName,

false

ref


        messageType);

08. return pInMsg;

09. }

We need to redecorate custom pipeline class with CategoryTypes.CATID_Any to be allowed add it to Pre-assemble component in receive pipeline and validate component in send pipeline .

The complete code of custom pipeline as following

001.using Microsoft.BizTalk.Component.Interop;

002.using Microsoft.BizTalk.Message.Interop;

003.using Microsoft.RuleEngine;

004.using System;

005.using System.Collections;

006.using System.Collections.Generic;

007.using System.Runtime.InteropServices;

008.using System.Xml;

009.using TechNetWiki.DynamicMappingResolver.Library;

010.

011.namespace VeriCash.BizTalk.PipelineComponents

012.{

013.

[ComponentCategory(CategoryTypes.CATID_Any)]

014. [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

015. [Guid("5153F77B-F6E3-4F6A-AB96-670D324C8F7A")]

016. public class


        DynamicMapResolverComp : IBaseComponent, IComponent, IComponentUI

017. {

018.

/// <summary>

019.

/// The
        cached sources to cache values of sources schema message types

020.

/// </summary>

021. private static


        Dictionary<

string

> cachedSources =

new


        Dictionary<

string

string>();

022.

023.

#region IBaseComponent
        Members

024.

025. public string


        Description

026. {

027. get

028. {

029. return "Dynamic mapping resolver";

030. }

031. }

032.

033. public string


        Name

034. {

035. get

036. {

037. return "Dynamic Mapping Resolver";

038. }

039. }

040.

041. public string


        Version

042. {

043. get

044. {

045. return "1.0";

046. }

047. }

048.

049. #endregion

050.

051.

#region IComponent
        Members

052.

053. public IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)

054.

055. string messageType = null;

056. string messageSourceType = pInMsg.Context.Read("MessageType"

"http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString();

057. string

mapName = GetMapStrongName(messageSourceType);

058.

System.IO.Stream
        originalStream = pInMsg.BodyPart.GetOriginalDataStream();

059.

pInMsg.BodyPart.Data
        = MapHelper.TransformStream(originalStream, mapName,

false

ref


        messageType);

060. return pInMsg;

061.

062.

063. #endregion

064.

065.

#region IComponentUI
        Members

066. public IntPtr Icon

067. {

068. get

069. {

070.

071. return System.IntPtr.Zero;

072. }

073.

074. }

075. public IEnumerator Validate(object obj)

076. {

077. return null;

078. }

079. #endregion

080.

081. #region Helper

082.

083. private string


        GetMapStrongName(

string messageType)

084. {

085. string mapStrongName = string.Empty;

086.

////create
        an instance of the XML object

087.

XmlDocument
        xmlDoc =

new


        XmlDocument();

088. xmlDoc.LoadXml(string

.Format(@"<ns0:BRERoot
        xmlns:ns0=

'http://TechNetWiki.DynamicMappingResolver.Schemas.BRESchema'>

089. <MessageType>{0}</MessageType>

090. <MapStrongName></MapStrongName>

091.

</ns0:BRERoot>",
        messageType));

092. if (cachedSources.ContainsKey(messageType))

093. {

094.

mapStrongName
        = cachedSources[messageType];

095. }

096. else

097. {

098.

////Prepare
        object to call BRE API

099.

TypedXmlDocument
        typedXmlDocument =

new TypedXmlDocument("TechNetWiki.DynamicMappingResolver.Schemas.BRESchema", xmlDoc);

100.

Microsoft.RuleEngine.Policy
        policy =

new


        Microsoft.RuleEngine.Policy(

"DynamicMappingResolverPolicy");

101. policy.Execute(typedXmlDocument);

102.

XmlNode
        messageTypeNode = typedXmlDocument.Document.SelectSingleNode(

"//MapStrongName");

103.

mapStrongName
        = messageTypeNode.InnerText;

104. policy.Dispose();

105.

////
        Fail if message type is unknown

106. if (string.IsNullOrEmpty(mapStrongName))

107. {

108. throw new


        Exception(

string.Format("Map for message type {0} is not exist", messageType));

109. }

110.

111.

cachedSources.Add(messageType,
        mapStrongName);

112. }

113. return mapStrongName;

114. }

115.

116. #endregion

117. }

118.}

Create a new BizTalk project and add assembly to <BizTalkInstallationFolder>\Pipeline Components
Add new item and choose Receive Pipeline
Right click to toolbox and choose dynamic mapping resolver as shown in figure 6

Figure 6. Adding Dynamic Mapping Resolver in ToolBox

Drag XML dis-assembler to disassemble part and drag Dynamic Mapping Resolver to Validate part as shown in figure 7.

Figure 7. Final Receive Pipeline

Dynamic map resolver in Outbound Maps

Add new item and select send pipeline.
Drag Dynamic Mapping Resolver to Pre-Assemble part and Xml-Assembler to Assemble part
as shown in figure 8.

Figure 8. Final Send Pipeline

Deploy BizTalk project

Note: that I am using xml instance for testing sample source code, if you want to use flat file then you can use Flat file dissembler and assembler.

Dynamic map resolver in Orchestration

You can have a sample orchestration as shown in figure 9

Figure 9. Sample orchestration

We can use dynamic transformation by using XLANG which exposes a transform method that can be called from within a Message Assignment shape inside of a Construct Message shape. This is the same method that is called when a Transform shape is used.

1.MapType=System.Type.GetType(msgBRE.MapStrongName);

2.transform(msgDestination) = MapType(msgSource);

Message / Variable	Identifier	Type	Comments
Message	msgBRE	TechNetWiki.DynamicMappingResolver.Schemas.BRESchema	The BRE instance message
Message	msgSource	System.Xml.XmlDocument	The incoming instance message
Message	msgDestination	System.Xml.XmlDocument	The outgoing instance message
Variablut	MsgType	System.Type	Type of map

Sample Code

You can find the source code belonging to this article on MSDN Code Gallery: BizTalk Server: Dynamic Mapping Resolver

Conclusion

In this article I explained and answered the following questions:

What is dynamic mapping resolver?
What are the problems that we can solve by using dynamic mapping resolver?
How to implement dynamic mapping resolver in receive/send pipeline and orchestration?

Creating Custom Adapter Provider in ESB Toolkit SFTP As an Example

2013-12-30T04:48:00.001-08:00

Introduction

ESB Toolkit "is a collection of tools and libraries that extend BizTalk Server capabilities of supporting a loosely coupled and dynamic messaging architecture. It functions as middleware that provides tools for rapid mediation between services and their consumers. Enabling maximum flexibility at run time, the BizTalk ESB Toolkit simplifies loosely coupled composition of service endpoints and management of service interactions." [MSDN]

ESB Toolkit Core Engine contains many components, one of the main component is Adapter Provider as shown in figure 1

Figure 1. ESB Toolkit Core Engine Component [MSDN]

"Adapter Providers are used in the BizTalk ESB Toolkit to set properties on outbound adapters. They provide a mapping between BizTalk ESB Toolkit configuration properties and Send Adapter context properties." [Flanders]

SFTP adapter is one of the new adapters added to the out of the box adapters of BizTalk Server 2013.

In this article I am going to explain how to create custom adapter provider using SFTP adapter as an example.

I supposed that you installed and configured BizTalk 2013 , ESB Toolkit 2.2 and prepared SFTP testing environment

Problem

There are a number of Adapter Providers included with the BizTalk ESB Toolkit 2.2 such as:

WCF-BasicHttp
WCF-Custom
WCF-WSHttp
FTP
File
SMTP
WebSphere MQ

However, how can I extend Adapter Provider Framework and use SFTP adapters?

Solution

The Adapter Provider framework provides a very easy way to extend it to include more BizTalk send adapters that are capable of dynamic configuration. These adapters are defined through the implementation of the IAdapterProvider interface within a .NET Framework assembly.

To create a custom SFTP adapter provider

"Step 1: Create an assembly that derives from the BaseAdapterProvider base class and contains a SetEndPoint method that sets the endpoint context properties of the message." [MSDN]

Add a class library project to you solution and give it a name like TechNetWiki.ESB.CommonAdapter.SFTPAdapter
Add a new class item to your class library project and name it AdapterProvider
Add references to the following assemblies
- %BTSINSTALLPATH%\Microsoft.BizTalk.Adapter.Sftp.dll
- %BTSINSTALLPATH%\\Microsoft.XLANGs.BaseTypes.dll
- %BTSINSTALLPATH%\\Microsoft.BizTalk.Pipeline.dll
- C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit\Bin\Microsoft.Practices.ESB.Adapter.dll
- C:\Windows\assembly\GAC_MSIL\Microsoft.BizTalk.GlobalPropertySchemas\3.0.1.0__31bf3856ad364e35\Microsoft.BizTalk.GlobalPropertySchemas.dll
The created class must implement BaseAdapterProvider add override AdapterName property and SetEndpoint method as show in the below code

01.namespace TechNetWiki.ESB.CommonAdapter.SFTPAdapter

02.{

03.    using Microsoft.Practices.ESB.Adapter;

04.    using Microsoft.XLANGs.BaseTypes;

05.    using System;

06.    using System.Collections.Generic;

07.    using System.Linq;

08.

09.    public class AdapterProvider : BaseAdapterProvider

10.    {

11.        public override string AdapterName

12.        {

13.            get

14.            {

15.                return "SFTP";

16.            }

17.        }

18.

19.        public override void SetEndpoint(Dictionary<string,string> resolverDictionary, XLANGMessage message)

20.        {

21.            if (resolverDictionary == null)

22.                throw new ArgumentNullException("resolverDictionary");

23.            if (message == null)

24.                throw new ArgumentNullException("message");

25.

26.            base.SetEndpoint(resolverDictionary, message);

27.

28.            try

29.            {

30.                string transportLocation = resolverDictionary["Resolver.TransportLocation"];

31.                string outboundTransportCLSID = resolverDictionary["Resolver.OutboundTransportCLSID"];

32.                string endpointConfig = resolverDictionary["Resolver.EndpointConfig"];

33.                string transportType = resolverDictionary["Resolver.TransportType"];

34.

35.                message.SetPropertyValue(typeof(BTS.OutboundTransportLocation), transportLocation);

36.                message.SetPropertyValue(typeof(BTS.OutboundTransportType), transportType);

37.                message.SetPropertyValue(typeof(BTS.OutboundTransportCLSID), outboundTransportCLSID);

38.

39.                if (!string.IsNullOrEmpty(endpointConfig))

40.                {

41.                    // parse delimited endpointconfig and set SFTP specific adapter properties

42.                    // endPointConfig data with this format "Key1=Value1;Key2=Value2;...."

43.                    var config = endpointConfig.Split(';').Select(part => part.Split('='))

44.                           .ToDictionary(split => split[0], split => split[1]);

45.                    // Set the context for the SFTP adapter

46.                    message.SetPropertyValue(typeof(SFTP.UserName), config["UserName"]);

47.                    message.SetPropertyValue(typeof(SFTP.Password), config["Password"]);

48.                    message.SetPropertyValue(typeof(SFTP.AccessAnyServerHostKey), config["AccessAnyServerHostKey"]);

49.                }

50.

51.            }

52.            catch (System.Exception ex)

53.            {

54.                throw;

55.            }

56.        }

57.    }

58.}
Right Click on project and select Properties -> Select Signing Tab-> check Sign the assembly -> Choose a strong name key file <New...> and name it like key.snk -> Build the project

"Step 2: Register the adapter provider by adding it to Esb.config configuration files using an <adapterProvider> element with a name for the adapter as the name attribute, the fully qualified name of the class as the type attribute, the moniker as the moniker attribute (multiple values should be separated by a comma), and optionally the assembly of the actual adapter as the adapterAssembly attribute." [MSDN]

We need to know the public key token of the created assembly to use it in esb.config file. Open Visual Studio Command Prompt -> Type sn -T <assemblyPath>
Open file C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit\esb.config
Add <adapterProvider name="SFTP" type="TechNetWiki.ESB.CommonAdapter.SFTPAdapter.AdapterProvider, TechNetWiki.ESB.CommonAdapter.SFTPAdapter, Version=1.0.0.0, Culture=neutral, PublicKeyToken=PublicKeyInStep1" moniker="SFTP" /> before </adapterProviders> tag and replace PublicKeyInStep1 with the generated public key token in step 1

"Step 3: Register the new assembly in the global assembly cache." [MSDN]

Open Visual Studio Command Prompt -> Type gacutil -i <assemblyPath>

Step 4: To let SFTP adapter be visible in Static Resolver Transport Name property

Create a new xml file and rename it SFTPPropertyManifest.xml in this path C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Microsoft.Practices.Services.Itinerary.DslPackage
Add the following script to your manifest file

<?xml version="1.0" encoding="utf-8" ?>

<adapterPropertyManifest adapterName="SFTP">

    <aliases>

        <alias name="globalPropertySchemas" value="Microsoft.BizTalk.GlobalPropertySchemas, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />

    </aliases>

    <properties>

        <property name="UserName" type="SFTP.UserName" description="The user name for the connection." encrypted="true" assembly="globalPropertySchemas" />

        <property name="Password" type="SFTP.Password" description="The password for the connection." encrypted="true" assembly="globalPropertySchemas" />

        <property name="AccessAnyServerHostKey" type="SFTP.AccessAnyServerHostKey" description="Determines if any SSH public host key fingerprint from the Server should be accepted." assembly="globalPropertySchemas" />

        <property name="EventArgs" type="System.EventArgs" assembly="mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

    </properties>

</adapterPropertyManifest>
Close visual studio

Test the solution

Use these steps to test the new custom sftp adapter provider

Open Visual Studio 2012
Create a new Class Library project and name it such as ESB.Itinerary.Library
Right click on ESB.Itinerary.Library project -> Add-> New BizTalk ESB Itinerary Designer -> name it StaticSftp.itinerary
Right click on surface of Itinerary Designer and set Model Exporter to Database Itinerary Exporter and Require Encryption Certificate to false As shown in figure 2

Figure 2: Itinerary Properties
From the Toolbox, drag an On-Ramp model element to the design surface. In the OnRamp1 Properties window, configure the following properties
- Click the Name property, and then type OnRamp.
- In the Extender drop-down list, click On-Ramp Extender.
- In the BizTalk Application drop-down list, click Microsoft.Practices.ESB.
- In the Receive Port drop-down list, click OnRamp.Itinerary.
as shown in figure 3

Figure 3: OnRamp Properties
From the Toolbox, drag an Off-Ramp model element to the design surface, and then place it to the right of the On-Ramp model element. In the OffRamp1 Properties window, configure the following properties:
- Click the Name property, and then type OffRamp.
- In the Extender drop-down list, click Off-Ramp ESB Extender.
- In the BizTalk Application drop-down list, click Microsoft.Practices.ESB.
- In the Send Port drop-down list, click DynamicResolutionOneWay.
as shown in figure 4

Figure 4: OffRamp Properties
From the Toolbox, drag an Itinerary Service model element to the design surface, and then place it to the right of the On-Ramp model element. In the ItineraryService1 Properties window, configure the following properties:
- Click the Name property, and then type StaticSftpSendPort.
- In the Itinerary Service Extender drop-down list, click Off-Ramp Extender.
- In the Off-Ramp drop-down list, select Send Handlers.
as shown in figure 5

Figure 5: StaticSftpSendPort Properties
Right-click the Resolver collection of the StaticSftpSendPort model element, and then click Add new Resolver. In the Resolver1 Properties window, configure the following properties:
- Click the Name property, and then type StaticSftpResolver.
- In the Resolver Implementation drop-down list, click Static Resolver Extension.
- In the Transport Name drop-down list, click SFTP.
- In the Transport Location set sftp://127.0.0.1:22//C/Users/SFTPUser/MessageTesting/OUT/Static_%MessageID%.xml
- In the Endpoint Configuration set AccessAnyServerHostKey=true&UserName=sftpuser&password=sftpuserpassword
as shown in figure 6
Note: You cannot update UserName and Password because we set encrypted attribute for both to true. To work around it we set the value directly.

Figure 6: StaticSftpResolver Properties
In the Toolbox, click Connector. Drag a connection from the OnRamp model element to the StaticSftpSendPort model element.Add another connection from StaticSftpSendPort to the OffRamp as show in figure 7

Figure 7: StaticSftp Itinerary diagram Properties
Right click on the surface of the Itinerary designer then click deploy
Open BizTalk Administration Console in Microsoft.Practices.ESB application-> Receive Locations->New->One-way Receive Location..->Select OnRamp.Itinerary receive port->Name it Rcv_File -> Configure ->Set Receive Folder -> Receive Pipeline ItinerarySelectReceivePassthrough as shown in figure 8

Figure 8: Creating Receive Location
Click on ... of ItinerarySelectReceivePassthrough ->Set ItineraryFactKey Resolver.Itinerary -> set ResolveConnectionString ITINERARY:\\name=StaticSftp; as shown in figure 9

Figure 9: Configure ItinerarySelectReceivePassthrough pipeline
Test the solution by adding file to the receive location path and check the sftp folder that configured in step 8

Sample Code

All of this sample can be found and downloaded in Microsoft Code Gallery:

BizTalk Server 2013: Creating Custom SFTP Adapter Provider in ESB Toolkit

Conclusion

In this article I illustrated how to create a custom adapter provider using the new out of the box SFTP adapter which was added to BizTalk 2013 as an example You can do the same steps for other custom adapters.

BizTalk Server:Using Global Variable in Mapping

2013-11-29T07:04:00.001-08:00

Introduction

Mapping is one of the most important artifacts in BizTalk Server. It allows us to perform, in a simple and visual manner, transformations between XML messages.
When you create a map ,as a BizTalk developer, you should make sure that you build it with a good and optimal solution.
Mapping sometimes could be one of the bottlenecks of your BizTalk solution.
In this article, we will explain the benefits of using global variable in one of the real-life scenarios .

Problem

As BizTalk developer, you found out a problem in the current solution that when you upload a file with 10,000 records, it takes more that 15 minutes to process this file.
After investigation, you observed that the bottleneck is in mapping artifact. the map is shown in figure 1.

Figure 1. Bad solution map

In this map, it is calling the external assembly to pick up a value from database.
In the first look, you may think that it is calling the external assembly for 1 time but it will call it for many times.
If you take a look to xsl file, you will find out that the calling external assembly is inside the for-each loop as shown in figure 2.

<?xml version="1.0" encoding="UTF-16"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0 ScriptNS0" version="1.0" xmlns:ns0="http://Biztalk.Mapping.GlobalVariable.Destination" xmlns:s0="http://Biztalk.Mapping.GlobalVariable.Source" xmlns:ScriptNS0="http://schemas.microsoft.com/BizTalk/2003/ScriptNS0">

  <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />

  <xsl:template match="/">

    <xsl:apply-templates select="/s0:Source" />

  </xsl:template>

  <xsl:template match="/s0:Source">

    <ns0:Destiation>

      <xsl:for-each select="SourceRepeatedRecords">

        <DestinationRepeatedRecords>

          <xsl:variable name="var:v1" select="ScriptNS0:GetLookupValue(string(../Key/text()))" />

          <Value>

            <xsl:value-of select="$var:v1" />

          </Value>

          <FieldA>

            <xsl:value-of select="FieldA/text()" />

          </FieldA>

          <FieldB>

            <xsl:value-of select="FieldB/text()" />

          </FieldB>

        </DestinationRepeatedRecords>

      </xsl:for-each>

    </ns0:Destiation>

  </xsl:template>

</xsl:stylesheet>

Figure 2. Xsl file of bad solution map

If we run debug view to check how many times it call external assembly for the source that contains 3 records.
We will find out it will call it 3 times as shown in figure 3.

Figure 3. Debug View Event logs output for bad solution

Now it is clear where is the problem. If we have 10,000 records in source schema , then it will hit a database table for 10,000 times.

Solution

We need to call the external assembly to pick up value from lockup table for 1 time instead of 10,000 times.
There are many solutions. One of these solutions is using a global variable.We can use a global variable to save a value within the first iteration and read the value from the global variable within the remain iterations.
as shown in figure 4.

Figure 4. Good Solution Map

In figure 4 , The functoid number 1 is the value mapping of the external assembly when iteration index is equal 1
In functoid number 7, we save value in a global variable and return it to value as shown in figure 5

Figure 5. Setting DB value in global variable

In functoid number 8 , we get the value of return the value of global variable which is in functoid number 6 when iteration is not equal 1.
functoid number 6 code is shown in figure 6

Figure 6. Get the value from global variable

If we test our solution, we will find out the external assembly was called only for 1 time instead of 3 times as shown in figure 7

Figure 7. Debug View Event logs output for bad solution

So, without using a global variable if we have 10,000 records in the source schema it will hit database table for 10,000 time.On the other hand, when we use the global variable,it will hit database table for 1 time.

Sample Code

You can find the source code belonging to this article at MSDN Code Gallery:BizTalk Server:Using Global Variable in Mapping

Conclusion

In this article we illustrated how to use the global variable in BizTalk mapper and explained how global variable can improve the performance of BizTalk solution.

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

BizTalk Server Dynamic Schema Resolver Real-life Scenario

2013-11-29T07:02:00.000-08:00

Introduction

As a BizTalk consultant you had an assignment that a service provider asked you to enhance the existing BizTalk solution.
You checked the current implementation and you found that the existing BizTalk solution as shown in Figure 1

Figure 1- Current BizTalk Solution Architecture

They are receiving different flat files from different customers and each customer has a different flat file format. All customers send these flat files in one receive path with a specific unique file name format ####YYYYMMDDHHMMSSNNN.txt

The first 4 digits are alphanumeric of customer code.

Each customer publishes a message to BizTalk Engine and other send ports subscribe these messages.

In this article, I will focus in dynamic resolver engine which is specifically in receive and send pipelines

Bad Solution

The implantation was any new customer added to the system, we need to do the followings steps:

Adding New Source Steps

Add a new flat file schema matching the structure of a new customer as shown in Figure 2

Figure 2- Sample Source Schema

Figure 1- Current BizTalk Solution Architecture
Add a new receive pipeline , add flat file disassembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 3

Figure 3- Receive Pipeline
Add a new receive location and set Receive Pipeline to the new receive pipeline that created in step 2 as shown in Figure 4

Figure 4- Set Receive Pipeline to receive location
Click on configure button and set File mask as shown Figure 5

Figure 5- Set File Mask

In case there is a new Destination then you need to do the following steps

Adding New Destination Steps

Add a new flat file schema matching the structure of a new destination as shown in Figure 6

Figure 6- Sample Destination Schema
Add a new send pipeline , add flat file assembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 7

Figure 7 - Send Pipeline
Add a new map between source and destination like sample as shown in Figure 8

Figure 8 - Sample Map between source and destination
Add a new Send Port in BizTalk Administration as show in Figure 9

Figure 9- Send Pipeline in Send port
Click configure and set file name format as show in Figure 10

Figure 10- Setting File Name Format
Select Outbound Maps and select the correct map as shown in Figure 11

Figure 11- Setting Outbound Map
Select Filters and Set BTS.ReceivePortName == Rcv_Messages which is the name of receive port as show in Figure 12

Figure 12- Setting Filter in send port

Problem

Now imagine that you want to add 100 customers as source then you need to add 100 of Receive Pipelines artifacts and you need to repeat steps of 2-4 of New Source Steps section 100 times. Another issue imagine that you want to add a new 100 destination then you need to create 100 of Send Pipelines artifacts and you need to repeat steps 2-7 of New Destination Steps section 100 times.

Good Solution

We need to implement a dynamic schema resolver which is a mechanism of dynamically associate message types to inbound instances based on an identifier contained in each instance message.
We need to build a custom pipeline that takes a message type from configuration database like Business Rule Engine or using your custom configuration database table and set Document Schema to dis-assembler using the unique key that distinguishes your source message, you can use context properties of the message or using any unique data of the content of the message.
In our case the unique key is the customer code which is the first 4 digits of the file name
For send custom pipeline we have the xml message inside BizTalk engine which contains the schema type, we just need to set document schema into assembler at runtime
Figure 13 shown the Good solution architecture

Figure 13- Good Solution Architecture using Dynamic Resolver Mechanism

We need to do the following implementation steps

Creating BRE Policy

Create new schema with 2 element one for Source Code and one for message type(Namespace#RootNode) as shown in Figure 14

Figure 14- BRE Schema
Create a new Policy SchemaResolverPolicy and add rule for each source as shown in Figure 15

Figure 15- Sample BRE Policy

Create Custom Pipeline Component

We need to create custom pipeline for receiving and sending pipeline as the following steps

Create Custom Receive Pipeline

Add a new class library project as shown in Figure 16

Figure 16- Add new class library to contain pipeline components
Add a signature to the assembly, to allow assembly to be put into the GAC. To do this, right-click the new project and choose Properties. Then go to the Signing tab, as shown in Figure 17

Figure 17- Adding a signature to the class library
To create classes that BizTalk recognizes as pipeline components, you need to implement some specific interfaces. These interfaces are specified in the Microsoft.BizTalk.Pipeline namespace. You therefore need to reference the assembly that contains this namespace and these interfaces. Right-click your project or on the References node in your project and click Add Reference. Go to the Browse pane and browse your way to <InstallationFolder>\Microsoft.BizTalk.Pipeline.dll and add this as a reference
To delegate call to Probe method which is implemented to investigate message to decide if message is recognizable or not. If message is recognizable then it returns “True” otherwise “False”. We need to add <InstallationFolder>\Microsoft.BizTalk.Pipeline.Components.dll.
To call BRE api we need to reference to <installation drive>:\Program Files\Common Files\ Microsoft.RuleEngine.dll
Delete the Class1.cs file that was automatically created for you and instead add a new class to your project by right-clicking the project and choosing Add, New Item. Choose Class as the type and provide a name for your class as show in Figure 18

Figure 18- Adding a new class to your class library.
Mark your class as a public and decorate it with some attributes Guid telling the pipeline designer how the component can be used in our receiving custom pipeline we need to use: CategoryTypes.CATID_PipelineComponent This category does not affect which stages a component can be placed in, but merely serves as an indication for the pipeline designer that this is a pipeline component.
CategoryTypes.CATID_DisassemblingParser Components in this category can be placed in the Disassemble stage of a receive pipeline and we assigning document schema in disassemble stage. To use the attribute you need to use namespace Microsoft.BizTalk.Component.Interop
as show in Figure 19

using Microsoft.BizTalk.Component.Interop;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    public class SchemaResolverFlatFileDasmComp

    {

    }

}

Figure 19- Adding Pipeline attributes
Decorate your class with GUID attribute by copy a new guid from TOOLS Create GUID As shown in Figure 20 and add namespace System.Runtime.InteropServices.

Figure 20- Create new GUID
All custom pipeline components must implement the following two interfaces:
- IComponentUI, which lets you specify an icon for the component and a method for validating any properties the user can set values in
- IBaseComponent, which basically lets you specify three properties that the pipeline designer uses to get a name, version, and description from your component to show in the designer As shown in Figure 21
using Microsoft.BizTalk.Component.Interop;

using System;

using System.Collections;

using System.Collections.Generic;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

    public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI

    {

        #region IBaseComponent Members

        public string Description

        {

            get

            {

                return "Flat file disassembler which resolves schemas using message context";

            }

        }

        public string Name

        {

            get

            {

                return "Receive Pipeline Schema Resolver";

            }

        }

        public string Version

        {

            get

            {

                return "1.0";

            }

        }

        #endregion

        #region IComponentUI Members

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }

        public IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

    }

}

Figure 21- Implement  IBaseComponent, IComponentUI interfaces
Implement interface IDisassemblerComponent which exposes two methods needed for a disassembler as shown in Figure 22

using Microsoft.BizTalk.Component;

using Microsoft.BizTalk.Component.Interop;

using Microsoft.BizTalk.Message.Interop;

using System;

using System.Collections;

using System.Collections.Generic;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

    public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent

    {

        #region IBaseComponent Members

        public string Description

        {

            get

            {

                return "Flat file disassembler which resolves schemas using message context";

            }

        }

        public string Name

        {

            get

            {

                return "Receive Pipeline Schema Resolver";

            }

        }

        public string Version

        {

            get

            {

                return "1.0";

            }

        }

        #endregion

        #region IComponentUI Members

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }

        public IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

        private FFDasmComp disassembler = new FFDasmComp();

        #region IDisassemblerComponent Members

        public IBaseMessage GetNext(IPipelineContext pContext)

        {

            // Delegate call to Flat File disassembler

            return disassembler.GetNext(pContext);

        }

        public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)

        {

            // Delegate call to Flat File disassembler

            disassembler.Disassemble(pContext, pInMsg);

        }

        #endregion

    }

}

Figure 22- Implement IDisassemblerComponent interface
Implement interface IProbeMessage which will implement Probe method which called, and the component is then executed only if the Probe method returns a value of True and this will do the functionality of schema resolver , We read ReceiveFileName context property from message context to read the first 4 digits of the file name to retrieve schema type from BRE as show in Figure 23

using Microsoft.BizTalk.Component;

using Microsoft.BizTalk.Component.Interop;

using Microsoft.BizTalk.Message.Interop;

using Microsoft.RuleEngine;

using System;

using System.Collections;

using System.Collections.Generic;

using System.IO;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

using System.Xml;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

[ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

[Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

public class


        SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent

{

private FFDasmComp disassembler = new FFDasmComp();

private static


        Dictionary<

string

> cachedSources =

new


        Dictionary<

string

string>();

#region IBaseComponent Members

public string


        Description

{

get

{

return "Flat file disassembler which resolves schemas using message context";

}

public string


        Name

{

get

{

return "Receive Pipeline Schema Resolver";

}

public string


        Version

{

get

{

return "1.0";

}

#endregion

#region IComponentUI Members

public IntPtr Icon

{

get

{

return System.IntPtr.Zero;

}

public IEnumerator Validate(object obj)

{

return null;

}

#endregion

#region IDisassemblerComponent Members

public IBaseMessage GetNext(IPipelineContext pContext)

{

// Delegate call to Flat File disassembler

return disassembler.GetNext(pContext);

}

public void


        Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)

{

// Delegate call to Flat File disassembler

disassembler.Disassemble(pContext, pInMsg);

}

#endregion

#region IProbeMessage Members

public bool


        Probe(IPipelineContext pContext, IBaseMessage pInMsg)

{

// Check arguments

if (null == pContext)

throw new


        ArgumentNullException(

"pContext");

if (null == pInMsg)

throw new


        ArgumentNullException(

"pInMsg");

// Check whether input message doesn't have a body part or it is set to null, fail probe in those cases

if (null == pInMsg.BodyPart || null == pInMsg.BodyPart.GetOriginalDataStream())

return false;

Stream sourceStream = pInMsg.BodyPart.GetOriginalDataStream();

string messageType = string.Empty;

//Read ReceivedFileName from the message context

string srcFileName = pInMsg.Context.Read("ReceivedFileName"

"http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();

srcFileName = Path.GetFileName(srcFileName);

//Substring the first four digits to take source code to use to call BRE API

string customerCode = srcFileName.Substring(0, 4);

//create an instance of the XML object

XmlDocument xmlDoc =

new


        XmlDocument();

xmlDoc.LoadXml(string

.Format(@"<ns0:Root
        xmlns:ns0=

'http://TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE'>

<SrcCode>{0}</SrcCode>

<MessageType></MessageType>

</ns0:Root>", customerCode));

//retreive source code in case in our cache dictionary

if (cachedSources.ContainsKey(customerCode))

{

messageType = cachedSources[customerCode];

}

else

{

TypedXmlDocument typedXmlDocument =

new


        TypedXmlDocument(

"TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE", xmlDoc);

Microsoft.RuleEngine.Policy policy =

new


        Microsoft.RuleEngine.Policy(

"SchemaResolverPolicy");

policy.Execute(typedXmlDocument);

XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MessageType");

messageType = messageTypeNode.InnerText;

policy.Dispose();

// Fail if message type is unknown

if (string.IsNullOrEmpty(messageType))

return false;

cachedSources.Add(customerCode, messageType);

}

// Get document spec from the message type

IDocumentSpec documentSpec = pContext.GetDocumentSpecByType(messageType);

// Write document spec type name to the message context so Flat File disassembler could access this property and

// do message processing for a schema which has document spec type name we've discovered

pInMsg.Context.Write("DocumentSpecName"

"http://schemas.microsoft.com/BizTalk/2003/xmlnorm-properties", documentSpec.DocSpecStrongName);

// Delegate call to Flat File disassembler

return disassembler.Probe(pContext, pInMsg);

}

#endregion

}

Figure 23- Implement IProbeMessage interface

Create Custom Send Pipeline

Add a new Item class to your class library project and rename it
Do same steps of creating custom receive pipeline class but for sending custom pipeline we need to use CategoryTypes.CATID_AssemblingSerializer attribute and implement IAssemblerComponent inteface as show in Figure 24

using Microsoft.BizTalk.Component.Interop;

using System;

using System.Runtime.InteropServices;

using Microsoft.BizTalk.Message.Interop;

using Microsoft.BizTalk.Component;

using System.Collections;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

[ComponentCategory(CategoryTypes.CATID_AssemblingSerializer)]

[Guid("ACF451CE-E60D-42E4-8AB0-ED4CADD8B23E")]

public class


        SchemaResolverFlatFileAsmComp : IBaseComponent, IComponentUI, IAssemblerComponent

{

System.Collections.Queue qOutputMsgs =

new


        System.Collections.Queue();

private FFAsmComp assembler = new FFAsmComp();

#region IBaseComponent members

public string


        Description

{

get

{

return "Flat file assembler which resolves schemas using message context";

}

public string


        Name

{

get

{

return "Send Pipeline Schema Resolver";

}

public string


        Version

{

get

{

return "1.0";

}

#endregion

#region IComponentUI Members

public IntPtr Icon

{

get

{

return System.IntPtr.Zero;

}

public IEnumerator Validate(object obj)

{

return null;

}

#endregion

#region IAssemblerComponent Members

public void


        AddDocument(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)

{

qOutputMsgs.Enqueue(pInMsg);

}

public IBaseMessage Assemble(IPipelineContext pContext)

{

IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();

assembler.AddDocument(pContext, msg);

return assembler.Assemble(pContext);

}

#endregion

}

Figure 24-Custom Send Pipeline component code

Create Receive and Send Pipeline artifacts

Build you Class library project and copy the dll file and paste it to this location <InstallationFolder>\Pipeline Components
Add new receive pipeline to your BizTalk project
Right Click on Toolbox then select choose items then choose as show in Figure 25

Figure 25- Selecting Custom Pipeline components
Drag Receive Pipeline Schema Resolver component from Toolbox and drop it to Disassemble part as shown in Figure 26

Figure 26- Drag Drop Receive Pipeline Schema Resolver component
Add new send pipeline to you BizTalk project
Drag Send Pipeline Schema Resolver component from Toolbox and drop it to assemble part as shown in Figure 27

Figure 27- Drag Drop Send Pipeline Schema Resolver component
Deploy You Pipeline BizTalk project then open BizTalk Administration and create new Receive location in your BizTalk application, configure the location and select the receive pipeline of schema resolver as shown in Figure 28

Figure 28-Select receive schema resolver pipeline
Create new send port, select file adapter ,configure the location and select send pipeline as shown in Figure 29

Figure 29-Select send schema resolver pipeline
Select all required Outbound maps to map files from source to destination as show in Figure 30

Figure 30-Select Outbound maps
Filter your send port to subscribe from receive port as show in Figure 31

Figure 31-Filter send port

We can realized now that if we want to add any new source or any new destination, you just need to add schemas, maps and configure BRE or your configuration table.
This means that you reduced the number of artifacts in your solution pipelines and receive and send ports.
In the bad solution if you have 100 sources and 100 destinations then you will have 100 receive locations, 100 send ports and 100 pipelines.
However, in the good solution you will have 1 receive location, 1 send port and 2 pipelines and your configuration table.

One more thing, that we can do more enhancement to our solution by passing parameter generically to BRE instead of depending on the first 4 charters of file name,
we can use any property of the message context but for demonstration purposes I just use the file name.
More over, we can use canonical schema and mapping (inbound and outbound mapping) dynamically which I will write another article about this subject later on.

Sample Source Code

You can find the sample source code in the following link sample source code

Conclusion

This article demonstrated

The real scenario of receiving and send Flat File from different sources with different formats and sending file to different destinations with different formats
How to implement this scenario using the static biztalk solution
How to implement the schema resolver mechanism using custom receiving and sending pipeline which made our solution more dynamic and easy to maintain
How to build custom receive and send pipeline in details
Benefits of using dynamic schema resolver mechanism
Call BRE API from C# code

How to Extract Email Attachments By Pipeline

2013-11-29T06:58:00.000-08:00

Introduction

In this article I will demonstrate step by step instructions of how to extract email attachments from POP3 adapter by using custom receive pipeline.

Problem Statement

You can have some customers that need to integrate with your integration middle-ware using POP3. So, They will send emails with multiple attachments and these attachments represent the messages that you need to process.

Solution

There are 2 BizTalk solutions that we can implement either in Message Level or Orchestration level. As performance perspective, a message level is much better because orchestration level has much persistence points which can reduce the performance of our integration solution.
If you would like to see the orchestration solution you can check this link.

To process email attachments through message level, then we need to process them using pipeline.
So, we need to implement custom receive pipeline and add it to disassemble part of receive pipeline
For now let's start our walk-though demonstration.

Create Custom Receive Pipeline

Create a new solution as shown in figure 1

Figure 1- Creating a new Solution
Create a new class library projects as shown in figure 2

Figure 2- Creating a new Class library project
Remove Class1.cs file
Create a new class as shown in figure 3

Figure 3- Creating a new class
Add Microsoft.BizTalk.Pipeline.dll from this path %BTSINSTALLPATH%
Add Microsoft.BizTalk.Pipeline.Components.dll as a reference from this path %BTSINSTALLPATH%\Pipeline Components
Implement the IBaseComponent, IComponentUI interfaces as shown in figure 4

namespace TechNetWiki.AttachmentsExtractor.PipelineComponent

{

    using System;

    using System.IO;

    using System.Text;

    using System.Reflection;

    using System.ComponentModel;

    using Microsoft.BizTalk.Message.Interop;

    using Microsoft.BizTalk.Component.Interop;

    using System.Runtime.InteropServices;



    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("01F04E58-7AD8-42EF-A3CB-D939B960E8F9")]

    public class AttachmentsExtractor : IBaseComponent, IComponentUI

    {

        #region IBaseComponent Members

        /// <summary>

        /// Gets Description of the component

        /// </summary>

        public string Description

        {

            get

            {

                return "Email Attachments Extractor";

            }

        }



        /// <summary>

        /// Gets Name of the component

        /// </summary>

        public string Name

        {

            get

            {

                return "EmailAttachmentsExtractor";

            }

        }



        /// <summary>

        /// Gets Version of the component

        /// </summary>

        public string Version

        {

            get

            {

                return "1.0";

            }

        }



        #endregion



        #region IComponentUI members

        /// <summary>

        /// Component icon to use in BizTalk Editor

        /// </summary>

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }



        /// <summary>

        /// The Validate method is called by the BizTalk Editor during the build

        /// of a BizTalk project.

        /// </summary>

        /// <param name="obj">An Object containing the configuration properties.</param>

        /// <returns>The IEnumerator enables the caller to enumerate through a collection of strings containing error messages. These error messages appear as compiler error messages. To report successful property validation, the method should return an empty enumerator.</returns>

        public System.Collections.IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

    }

}

Figure 4-Implementing IBaseComponent, IComponentUI
Implement IDisassemblerComponent interface which is mandatory to implement GetNext and Disassemble methods
- GetNext method returns messages resulting from the disassemble method execution
- Disassemble method used to split or process the incoming message document
When we receive an email we will receive it as a message associated with multi-parts messages so In Disassemble method we need to loop in each Part of the message and create a new message.We will will start looping from index 1 because index zero is the body content which we are not interested as shown in figure 5

#region IDisassemblerComponent members

        /// <summary>

        /// Returns messages resulting from the disassemble method execution

        /// </summary>

        /// <param name="pc">the pipeline context</param>

        /// <returns></returns>

        public IBaseMessage

            GetNext(IPipelineContext pc)

        {

            // get the next message from the Queue and return it

            IBaseMessage msg =null;

            if ((_msgs.Count > 0))

            {

                msg = ((IBaseMessage)(_msgs.Dequeue()));

            }

            return msg;

        }

        /// <summary>

        /// called by the messaging engine when a new message arrives

        /// </summary>

        /// <param name="pc">the pipeline context</param>

        /// <param name="inmsg">the actual message</param>

        public void Disassemble(Microsoft.BizTalk.Component.Interop.IPipelineContext pc, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)

        {

            var partName =string.Empty;

            // we start from index 1 because index zero contain the body of the message

            // which we are not interested

            for (int i = 1; i < inmsg.PartCount; i++)

            {

                IBaseMessagePart currentPart = inmsg.GetPartByIndex(i,out partName);

                Stream currentPartStream = currentPart.GetOriginalDataStream();

                var ms =new MemoryStream();

                IBaseMessage outMsg;

                outMsg = pc.GetMessageFactory().CreateMessage();

                outMsg.AddPart("Body", pc.GetMessageFactory().CreateMessagePart(),true);

                outMsg.Context = inmsg.Context;

                currentPartStream.CopyTo(ms);

                string attachmentContent = Encoding.UTF8.GetString(ms.ToArray());

                MemoryStream attachmentStream =new System.IO.MemoryStream(

                System.Text.Encoding.UTF8.GetBytes(attachmentContent));

                outMsg.GetPart("Body").Data = attachmentStream;

                outMsg.Context.Write("ReceivedFileName","http://schemas.microsoft.com/BizTalk/2003/file-properties", partName);

                _msgs.Enqueue(outMsg);

            }

        }

        #endregion

Figure 5- Implementing IDisassemblerComponent interface

the Full code of custom pipeline as show in figure 6

namespace TechNetWiki.AttachmentsExtractor.PipelineComponent

{

    using System;

    using System.IO;

    using System.Text;

    using System.Reflection;

    using System.ComponentModel;

    using Microsoft.BizTalk.Message.Interop;

    using Microsoft.BizTalk.Component.Interop;

    using System.Runtime.InteropServices;

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("01F04E58-7AD8-42EF-A3CB-D939B960E8F9")]

    public class AttachmentsExtractor : IBaseComponent, IComponentUI, IDisassemblerComponent

    {

        #region IBaseComponent Members

        /// <summary>

        /// Gets Description of the component

        /// </summary>

        public string Description

        {

            get

            {

                return "Email Attachments Extractor";

            }

        }

        /// <summary>

        /// Gets Name of the component

        /// </summary>

        public string Name

        {

            get

            {

                return "EmailAttachmentsExtractor";

            }

        }

        /// <summary>

        /// Gets Version of the component

        /// </summary>

        public string Version

        {

            get

            {

                return "1.0";

            }

        }

        #endregion

        #region IComponentUI members

        /// <summary>

        /// Component icon to use in BizTalk Editor

        /// </summary>

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }

        /// <summary>

        /// The Validate method is called by the BizTalk Editor during the build

        /// of a BizTalk project.

        /// </summary>

        /// <param name="obj">An Object containing the configuration properties.</param>

        /// <returns>The IEnumerator enables the caller to enumerate through a collection of strings containing error messages. These error messages appear as compiler error messages. To report successful property validation, the method should return an empty enumerator.</returns>

        public System.Collections.IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

        /// <summary>

        /// this variable will contain any message generated by the Disassemble method

        /// </summary>

        private System.Collections.Queue _msgs = new System.Collections.Queue();

        #region IDisassemblerComponent members

        /// <summary>

        /// Returns messages resulting from the disassemble method execution

        /// </summary>

        /// <param name="pc">the pipeline context</param>

        /// <returns></returns>

        public IBaseMessage

            GetNext(IPipelineContext pc)

        {

            // get the next message from the Queue and return it

            IBaseMessage msg =null;

            if ((_msgs.Count > 0))

            {

                msg = ((IBaseMessage)(_msgs.Dequeue()));

            }

            return msg;

        }

        /// <summary>

        /// called by the messaging engine when a new message arrives

        /// </summary>

        /// <param name="pc">the pipeline context</param>

        /// <param name="inmsg">the actual message</param>

        public void Disassemble(Microsoft.BizTalk.Component.Interop.IPipelineContext pc, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)

        {

            var partName =string.Empty;

            // we start from index 1 because index zero contains the body of the message

            // which we are not interested

            for (int i = 1; i < inmsg.PartCount; i++)

            {

                IBaseMessagePart currentPart = inmsg.GetPartByIndex(i,out partName);

                Stream currentPartStream = currentPart.GetOriginalDataStream();

                var ms =new MemoryStream();

                IBaseMessage outMsg;

                outMsg = pc.GetMessageFactory().CreateMessage();

                outMsg.AddPart("Body", pc.GetMessageFactory().CreateMessagePart(),true);

                outMsg.Context = inmsg.Context;

                currentPartStream.CopyTo(ms);

                string attachmentContent = Encoding.UTF8.GetString(ms.ToArray());

                MemoryStream attachmentStream =new System.IO.MemoryStream(

                System.Text.Encoding.UTF8.GetBytes(attachmentContent));

                outMsg.GetPart("Body").Data = attachmentStream;

                outMsg.Context.Write("ReceivedFileName","http://schemas.microsoft.com/BizTalk/2003/file-properties", partName);

                _msgs.Enqueue(outMsg);

            }

        }

        #endregion

    }

}

Figure 6- Custom pipeline full code
Build the project then copy the generated TechNetWiki.AttachmentsExtractor.PipelineComponent.dll to %BTSINSTALLPATH%\Pipeline Components

Create Receive Pipeline

Create a new BizTalk Project by right click solution and select new project then select Empty BizTalk Server Project and name it TechNetWiki.AttachmentsExtractor.Pipelines as shown in figure 7

Figure 7- Create Pipeline BizTalk Project
Right click TechNetWiki.AttachmentsExtractor.Pipelines project -> Select Add-> New Item -> Select Receive Pipeline-> Name it AttachmentsExtractorPipeline as shown in figure 8

Figure 8- Create a new receive pipeline
Right click to ToolBox window ->select Choose Items... -> select BizTalk Pipeline Components -> check EmailAttachmentsExtractor as shown in figure 9

Figure 9- Add pipeline component to ToolBox
Drag drop EmailAttachmentsExtractor to Disassemble part as shown in figure 10

Figure 10- Setting pipeline component in Disassemble part
Right click project TechNetWiki.AttachmentsExtractor.Pipelines -> select Properties -> select Sign tab-> check Sign the assembly -> choose a strong name key file -> select New and rename it to key.snk -> select Deployment tab -> set Application Name to TechNetWiki.AttachmentsExtractor
Right click TechNetWiki.AttachmentsExtractor.Pipelines -> select Deploy

Configure POP3 Adapter

Open BizTalk Server Administration Console
Right click TechNetWiki.AttachmentsExtractor - Receive Port -> select New -> One-way Receive Port... -> change name to Rcv_EmailTest -> select Receive Location tab -> change name to RcvLoc_EmailTest
Select POP3 adapter-> click Configure button -> set POP3 adapter configure as shown in figure 11 In this demo I am using my hotmail account to configure POP3 adapter

Figure 11 - Configuring POP3 adapter
Create a new send port and name it Snd_EmailTest -> select File Adapter and configure it as shown in figure 12
figure 12- Creating send port
Select Filters tab then set filter to our receive port name as shown in figure 13

Figure 13- Setting filter

Test the Solution

Send email with sample attachment message to the configured email that we configured on POP3 adapter which is my demo YouUserName@hotmail.com
Check send port location

Sample Code

You can find the sample source code in the following link

Conclusion

In BizTalk Server there are two ways to extract email attachments either using custom receive pipeline or orchestration

In this article, I walked-through in step by step how to create a custom receive pipeline to extract email attachments.

An Easy Way to Get BizTalk Alerts

2013-11-29T06:51:00.000-08:00

Introduction

As a BizTalk administrator, you need to know about any issues in your BizTalk environment. It is crucial to know about these issues before they become real problems or cause any unnecessary outage for any integration services provided by your BizTalk server.

In this article I will demonstrate an easy way to send alerts in case there are any errors or warnings in the BizTalk environment.

Problem Statement

You may have some customers connecting to your integration middle-ware through receive adapters such as POP3, SFTP, FTP, File, etc. In some cases you may get some warnings and errors which affect these receive adapters and make them disabled once the error threshold is exceeded.

In many instances you would not be able to use the BizTalk Administration Console for alerting as these entries do not even register in the BizTalk Message Box.

Therefore, it is very important to be notified before the end users prompted with warning or error messages when they are trying to upload into integration middle-ware. It is equally important to take a proactive approach to resolve any upcoming issues you might be experiencing with your BizTalk server.

Solution

You might be attempted to use monitoring tools like SCOM or BizTalk360,..These tools are not free and some companies might find them costly to licence and implement.

If your project does not have a budget to deploy these motioning tools, there is an easy way to get BizTalk alerts by using a combination of PowerShell scripting and the built-in Windows Task Scheduler through the following steps:

Select Windows Start Button ->Control Panel->Administrative Tools -> Task Scheduler as shown in Figure 1 or use the short cut command in run window control schedtasks

Figure 1 - Accessing Task Scheduler
In Task Scheduler window select Microsoft folder->click on create task -> in General Tab Name the task as shown in Figure 2

Figure 2- Create Task
Select Trigger Tab->Click New button-> in the Begin the task select on an event ->select custom->click New Event Filter button-> in Event Sources select BizTalk -> In case you want to filter by using event id you can set event id as shown in Figure 3

Figure 3- Create Trigger
Select Action Tab->Click New Button->In Action select Send an e-mail->set email settings properties as shown in Figure 4

Figure 4 - Create Exchange Email Action
If you want to to add attachment of event logs to be included in email, then you need to create a new action to run the following command which write your filter log into a text file in my case I filtered on EventId 5740 which is pop3 warning that happens once pop3 user not authenticated and save it in this path C:\SendEmail\temp\logs.txt

del C:\SendEmail\temp\logs.txt

wevtutil qe Application"/q:*[System [(EventID=5740)]]" /f:text /rd:true /c:1> C:\SendEmail\temp\logs.txt
Save command into cmd file and rename it into path C:\SendEmail\WriteLog.cmd
Create new action -> In Action field select start a program-> Set Program/Script Path as shown in Figure 5

Figure 5 - Create Command Action
Sort action and make sure that running the command action in the first action as shown in Figure 6

Figure 6- Sort actions
In case you do not want to use exchange mail server, then you can write the following powershell script and save it in this path C:\SendEmail\SendEmailScript.ps1

$emailSmtpServer="smtp.gmail.com"

$emailSmtpServerPort="587"

$emailSmtpUser="yourusername@gmail.com"

$emailSmtpPass="yourpassword"

$emailFrom="fromusername@gmail.com"

$emailTo="sendtousername@gmail.com"

$emailMessage=New-Object System.Net.Mail.MailMessage( $emailFrom , $emailTo )

$emailMessage.Subject="BizTalk Error"

$emailMessage.IsBodyHtml=$true

$emailMessage.Body=@"

<p>There is a BizTalk issue</p>

<p>Check the attachment</p>

"@

$emailAttachment=New-Object System.Net.Mail.Attachment("C:\SendEmail\temp\logs.txt")

$emailMessage.Attachments.Add($emailAttachment)

$SMTPClient=New-Object System.Net.Mail.SmtpClient( $emailSmtpServer , $emailSmtpServerPort )

$SMTPClient.EnableSsl=$true

$SMTPClient.Credentials=New-Object System.Net.NetworkCredential( $emailSmtpUser , $emailSmtpPass );

$SMTPClient.Send( $emailMessage )
Create another command file to run powershell script as below and save it in this path C:\SendEmail\RunPowerShellScript.cmd

powershell C:\SendEmail\SendEmailScript.ps1

Now SendEmail folder should be as shown in Figure 7

Figure 7- Files inside SendEmail Folders
Update Send Email Action which we created in step 4 to run a program with path C:\SendEmail\RunPowerShellScript.cmd as shown in
Figure 8- Edit Action to use PowerShell script
To test your work, create pop3 receive location and set a wrong username and password

Conclusion

In this article, I walked-through in step by step how to use an easy way to send any BizTalk alerts to specific email using a built-in Windows Task Scheduler .

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

How to Configure SFTP Send Port Dynamically in BizTalk Server 2013

2013-11-29T06:43:00.002-08:00

Introduction

SFTP Adapter is a new adapter added to the built-in adapters in BizTalk Server 2013. I explained how to use SFTP adapter in article BizTalk Server 2013: How to use SFTP Adapter. In that article I explained how to prepare SFTP testing environment and how to configure SFTP adapter in design time. In this article I will walk-through in how to configure SFTP send port dynamically at run-time.

Problem

You have an orchestration that send messages to a specific customer via SFTP adapter. However, your integration middleware start to be expanded and start to have multiple types of customers that need to receive messages in different SFTP addresses. So, how we can update our orchestration to enable it to send messages to dynamic addresses.

Solution

We need to create a dynamic send port and configure it at run time.Dynamic ports allow the physical location of a physical send port (one-way or solicit-response) to be determined at run time. The ability to specify the transport protocol and address at run time allows for the flexibility of routing messages depending on message content or message context. We can configure the send port to be a dynamic within the orchestration and we can save and retrieve properties of SFTP adapter configuration information from our configuration DB or Business Rule Engine (BRE)

Step By Step Implementation

Create a source schema (CustomerInfo) as shown in figure 1

Figure 1. CustomerInfo Schema Structure
Create a schema for BRE settings as shown in figure 2, in my demo I just want Address, username and password we can add more if you want to use digital certificate

Figure 2. Sftp Bre Schema Structure
By using Business Rule Composer create the SFTP Dynamic Policy and rule for each customer as shown in Figure 3, 4, 5 then Publish and Deploy policy

Figure 3. Customer A Rule

Figure 4. Customer B Rule

Figure 5. Customer C Rule
Create an orchestration that processes a message and send it to a dynamic port, I created a sample orchestration as show in figure 6

Figure 6. Business Process Orchestration

Define the following variable and messages as the following table

Message / Variable	Identifier	Type	Comments
Message	msgIn	TechNetWiki.DynamicSftpPort.CustomerInfo	The incoming message
Message	msgOut	TechNetWiki.DynamicSftpPort.CustomerInfo	The outgoing message
Message	msgSftpBre	TechNetWiki.DynamicSftpPort.SftpBreSchema	The Sftp Bre info message
Variable	varSftpBreXmlDoc	System.Xml.XmlDocument	A variable to load msgSftpBre message

Inside Assign_SftpBre assignment shape we need create xml document as the following code

varSftpBreXmlDoc.LoadXml(@"<ns0:SftpInfo xmlns:ns0='http://TechNetWiki.DynamicSftpPort.SftpBreSchema'>

  <Address></Address>

  <UserName></UserName>

  <Password></Password>

  <CustomerId></CustomerId>

</ns0:SftpInfo>");

msgSftpBre = varSftpBreXmlDoc;

msgSftpBre.CustomerId=msgIn.CustomerId;
Inside AssignDynamicSftpInfo assignment shape we will set the values retrieved from BRE to fill the msgOut and DynamicSFTPPort properties

msgOut=msgIn;

msgOut(SFTP.UserName)=msgSftpBre.UserName;

msgOut(SFTP.Password)=msgSftpBre.Password;

msgOut(SFTP.AccessAnyServerHostKey)=true;

DynamicSFTPPort(Microsoft.XLANGs.BaseTypes.Address)=msgSftpBre.Address +"%MessageID%.xml";
Create a send port in configuration wizard Set Port binding to Dynamic as shown in figure 5

Figure 7. Set Dynamic Port Binding
Deploy your project and create file receive port/location then configure orchestration then enlist and start the application.
Test the solution by adding some sample files with different customer id.In this demo I created three folders A,B,C inside the SFTP server to test the output messages.
You can check BizTalk Server 2013: How to use SFTP Adapter to know how to prepare sftp testing environment

Sample Code

You can find the source code belonging to this article at MSDN Code Gallery:BizTalk Server 2013: How to configure SFTP Send Port Dynamically

Conclusion

In this article, we showed how to create a dynamic sftp send port inside orchestration and how to configure it at run time by using Business Rule Engine.

How to use SFTP Adapter using BizTalk Server 2013

2013-11-27T04:01:00.001-08:00

Introduction

BizTalk server has supported the FTP (File Transfer Protocol) adapter since its early release.
With the BizTalk 2010 release, Microsoft enhanced the existing FTP adapter to support SSL (FTPS), which encrypts the data transfers between systems. In the BizTalk 2013 release, Microsoft added SFTP (SSH File Transfer Protocol also called Secure File Transfer Protocol). This has been much anticipated over the years and it is great that a true SFTP adapter now exists with BizTalk server. In the past for one had to rely on the open source adapter developed by Blogical found on CodePlex or an adapter from commercial third parties (like /n software SFTP). Now you will have advantages of Microsoft support for out-of-the-box of SFTP adapter.
For more information of BizTalk Server 2013 new features check this link
http://www.microsoft.com/en-us/biztalk/whats-new.aspx
In this article I will demonstrate step by step of the followings:

How to prepare a testing environment to communicate with the SFTP adapter as Figure 1?
How to configure BizTalk SFTP Adapter?
How to test BizTalk Adapter?

Figure 1. Demo Architecture

How to prepare a testing environment to communicate with the SFTP adapter?

Create a new local user Control Panel -> Administrative Tools -> Computer Management -> Local Users and Groups -> right click User -> New User-> Specify a User name and password.
Switch and Login for one time using SFTPUser to create user related folder and files.
Download Windows SFTP server tool that allows you to connect, retrieve and put files to try out the functionality of the SFTP Adapter.There are many tools For this demo I am using the following one http://dl.bitvise.com/BvSshServer-Inst.exe ,Then install it with default settings and restart your machine and make sure that Bitvise SSH Server service is started.
Download any SFTP client tool in this demo I am using the following tool http://winscp.net/download/winscp517setup.exe .Then install it with default settings and restart your machine.
Open WinSCP and enter Host Name , username and password as figure 2 then click login button then click yes in warning dialog .

Figure 2. WinSCP Login Dialog .

Now you will see the following window as Figure 3.

Figure 3. WinSCP Start-up window
Create a new Directory by selecting Files-> New -> Directory as figure 4.

Figure 4.Creating new Folder in WinSCP
In create Folder window enter folder name in this demo I named it MessageTesting and check Set permissions as figure 5.

Figure 5.Set Folder Permissions in WinSCP
Create folder IN and OUT inside folder MessageTesting same settings of Step 7 and the path for receive location should be as figure 6.

Figure 6.Path of IN Folder in WinSCP

Now we are ready to configure the BizTalk SFTP Adapter.

How to configure BizTalk SFTP Adapter?

Create a receive port I name it Rcv_SFTP then create receive location Rcv_SftpLoc then choose type of receive location to SFTP from drop down list then click Configure then change AcceptAnySSHServerHostKey property to True then fill other settings as Figure 7.

Figure 7.Configuring Receive SFTP Adapter

You can copy path of sftp folder to clipboard as figure 8.

Figure 8.Copy Path to Clipboard in WinSCP
Create Send port and Name Snd_SFTP select type from drop down list SFTP then click Configure then fill setting as Figure 9.

Figure 9.Send SFTP Adapter Configuration
Configure Filter of send port subscribe messages from port Rcv_SFTP as figure 10.

Figure 10.Filter SFTP send port Configuration
Enable receive location and Start send port.

How to test BizTalk Adapter?

Copy any file to IN folder path by drag drop from left window panel to right panel as Figure 11.

Figure 11.Copy dialog window in WinSCP
Check OUT folder for generated file as Figure 12

Figure 12.Copy dialog window in WinSCP

Conclusion

In this demonstration, we investigated:

The receive and send SFTP adapter, which is a new Feature comes with BizTalk Server 2013.
The step by step of preparation of an environment for testing the transfer of messages using the SFTP adapter.
And how to configure and test SFTP adapter.

Receive and Send Excel Files from Existing Templates using BizTalk Server

2013-11-27T03:51:00.000-08:00

Introduction

As a service provider that we provide a service to multiple customers and channels.
There is a new requirement that we are expecting to receive from each customer a specific excel file and sending a different excel file.
You can imagine that we could receive hundreds of different excel files from customers.
Also, we have an integration point for each customer that we need to map the customer format file to our canonical schema.

BizTalk Solution Approach

We need to build two pipelines one for receiving and one for sending excel files.
For receiving we need to parse an excel file using the configured template file to generate an xml file matching an existing customer schema as demonstrated in Figure 1

Figure 1. Receiving Excel Pipeline Diagram

For sending, we need to convert xml file to generate an excel file via custom sending pipeline using an existing excel template as demonstrated in figure 2

Figure 2. Sending Excel Pipeline Diagram

What is the best excel API to be used?

There are some of free excel APIs that you can use to process excel file like ExtremeML, Microsoft Excel Object Library, epplus and NPOI.I read a lot of articles and reviews and I tried to do some of POCs for eachI concluded that the best API I can be used and support both xls and xlsx extensions with the best performance is by using NPOI API.
You can find the api in this link http://npoi.codeplex.com

How to Prepare Excel template file?

I will give one example how to prepare a template file from a sample of customer excel file.

Let's say we got the file from customer A as Figure 3

Figure 3. Sample Excel Document

We need to know where the location of repeated records and where are the other fields.
First we need to prepare the template manually by making a place holder for each column name of repeated records and a label as a comment (DATAROW) where we can find the first repeated record then the place holders of other fields like figure 4

Figure 4. Preparing Excel Template Document

Then we need to generate a schema depending on a template file that we build it on figure 4
I build a simple tool to generate a schema from an existing template as Figure 5 or you can create schema manually

Figure 5. Tool to generate xsd from Existing Template

The tool will generate the schema as Figure 6

Figure 6. Generated xml scehma

How to configure the excel processor custom pipeline?

I checked NPOI api examples and I built my own custom pipeline component ExcelProcessor.BizTalk.PipelineComponents.dll and I copy it to this folder %BTSInstallPath%/Pipeline Components and you need to add any dependents dlls to GAC like the following dlls:

ICSharpCode.SharpZipLib.dll
NPOI.dll
NPOI.OOXML.dll
NPOI.OpenXml4Net.dll
NPOI.OpenXmlFormats.dll

for more information check Developing Custom Pipeline Components

Then you need to add a new receive and send pipeline then right click to Toolbox then select choose items and select as Figure 7

Figure 7. Adding excel decoder and encoder

Open Your receive pipeline then add ExcelPipelineDecoder from ToolBox and drag it to Decode part as Figure 8

Figure 8. Excel Receive Pipeline

Open Your send pipeline then add ExcelPipelineEncoder from ToolBox and drag it to Encode part as Figure 9

Figure 9. Excel Send Pipeline

Now your pipelines are ready to be deployed. Deploy your BizTalk project then open BizTalk Administration to configure pipelines
For testing purposes, I just added CustomerA.xsd and receive and send pipelines as Figure 10

Figure 10. BizTalk Solution Explorer

Now I will create one receive port as Rcv_Excel and a receive location as Rcv_ExcelLoc
and I will configure the Rcv_ExcelLoc as figure 11

Figure 11. Configuring receive port

Then we need to configure path of the template and TargetNameSpace of schema that related to excel template for ExcelReceivePipeline as figure 12

Figure 12. Configuring custom receive pipeline properties

for testing purposes I created a send port with XmlTransmit Send Pipeline and filtered it to BTS.ReceivePortName==Rcv_Excel
and it generates the following output as figure 13

Figure 13. Generated xml file

Conclusion

In this article I demonstrated the BizTalk approach for receiving and sending excel documents using existing templates for multiple customers.

You can use this approach for building BizTalk pipelines from existing templates for Excel, pdf ,word and any type of templates.

The challenge is how to work and deal with APIs then this approach will be easy to maintain and reuse.

The simplest way to GAC your project from IDE.

2009-12-30T01:12:00.000-08:00

As a BizTalk Developer you always need to add your dlls project to Global Assembly Cashe.
The simplest way to do this procedure is by a command which you need to create it in IDE.
By doing the following procedures:
1-Open VS.NET.
2- In Tool bar click Tools
3- From Tools click External Tools..
4-In External Tools dialog box click Add button.
5-In Title field add title like Install to GAC.
6- In Command field browse a gacutil.exe file which is in
C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin\gacutil.exe
it depends where you install your visual studio
7-In argument filed add the following /f /i $(TargetPath)
i means install to assembly
f means if dll exsits then overwite it
$(TargetPath) is the path if selected project
8-In initial directory field add $(TargetDir)
$(TargetDir) means directory of the selected project
9- Check Use Output window to inform you in output window the dll has been added to GAC
10- Click Ok button.

Now your cammand is ready.

Just select the project you want to add it to GAC
then select tools then Install to CAG command you created.

Hope this will Help!

How to use DebugView tool ?

2009-12-15T06:21:00.000-08:00

When I was developing BizTalk Applications, I used to work with EvenLog command to trace BizTalk work flow actions in Development Phase.

Then in deployment phaseI go through all EventLog commands to remove them.

DebugView is a cool tool is provided from microsoft and it needs some simple steps to use it, I will show you a simple example how to use it:

1- Download the tool from http://www.microsoft.com/technet/sysinternals/Miscellaneous/DebugView.mspx

2- Extract the zip file and run Dbgview.exe

3- I created a simple windows application with button and I added the following to button event.

System.Diagnostics.Debug.WriteLine("test -" + " Debug Suleiman");
System.Diagnostics.Trace.WriteLine("test -" + " Trace Suleiman");

4- I ran the application in debug mode then I clicked the button and I found following in DebugView.exe window

5- If you want to filter the view and highlight your required event, you have to click edit then Filter/Highlight or ctrl+L shortcut.

6- In this example I included all (*) , exclude empty and Trace in green highlight in Filter1 & Debug in purple highlight as the following:

7- If I convert my application to release mode you and click button again, will find that just Trace will be viewed

Conclusion:

Event logging uses disk space and normally takes processor time away from applications. So do not store unimportant information or large quantities of data in the event log. Generally, an event log should be used to log unexpected conditions or errors and not user actions or performance tracking information.

use System.Diagnostics.Debug.WriteLine("") will be viewed in Debug mode
Debug command does not work with Release Mode
System.Diagnostics.Trace.WriteLine will be viewed in both Release & Debug modes

You can use DebugView to check your required actions and you don't need to remove your trace lines in deployment phase. and your actions will be viewed by this cool tool

There are filter option and you can import/export your trace logs

Unable to run Biztalk 2009 Project using Visual Studio 2008

2009-12-15T04:39:00.000-08:00

I was trying to run a BizTalk Project using Visual Studio 2008 but I got the following error message:

The project file btproj cannot be opened.
The project type is not supported by this installation.

Causes:

BizTalk Developer tools and SDK was not installed

Solution:
You need to modify the the BizTalk 2009 installation and check Developer tools & SDK option

I hope this will help :)

How to install BizTalk 2009 in Multicomputer environment?

2009-11-18T00:56:00.000-08:00

It took me 3 days to install BizTlak 2009 in Multicomputer environment because I faced many problems to install.
I think if you follow the following steps, you will need about a day to finish installation.

1- You need to ask customer to prepare a domain windows user
and the following windows domain groups:
-SSO Administrators
-SSO Affiliate Administrators
-BizTalk Server Administrators
-BizTalk Server Operators
-BizTalk Application Users
-BizTalk Isolated Host Users
2- Install Sql Server 2008 in the database server
3- Install Sql Server 2008 client tool in BizTalk Server
4- Install DTC service in BizTalk server and Database Server.
5- Configure DTC to allow remote client in BizTalk & Database servers
6- Now you are ready to install BizTlak server 2009 software
7- You have to cinfigure databases of BizTalk with the domain users & groups which are created before.

How to Add BizTalk Project to Team Foundation Server TFS?

2009-05-06T02:04:00.000-07:00

Unfortunately, you can't create biztalk project to TFS .
Once I decided to try a tricky work around solution which is to add existing biztalk project to TFS solution.Then, you can use all TFS tools with your BizTalk Project.
I tried it and I don't believe myself how I didn't try it ages ago.

I hope you get benifits from my tricky soltuion

BizTalk 2006 R2 Deep Dive Training Cairo - Egypt

2008-05-15T04:25:00.000-07:00

How to add applicationSettings to your BizTalk solution?

2008-05-13T00:33:00.000-07:00

I got stuck when I wanted to add
my applicationSettings of my assembly to my BizTalk solution but I caught the solution
and I am very glad to share the solution.

First you have to copy your app.config
without
<?xml version="1.0" encoding="utf-8" ?>

like this

</section>

</sectiongroup>

<mycompany.projectname.properties.settings>

</setting>

</setting>

<value>False</value>

</setting>

</setting>

</mycompany.projectname.properties.settings>

</applicationsettings>

</configsections>

Then paste it before tag </configuration>
in the

C:\Program Files\Microsoft BizTalk
Server 2006\BTSNTSvc.exe.config
file

After that, restart the BizTalk
host instance service.

That's it.

I hope you get the solution. I will
be very glad to receive any comments

BizTalk Mapping Tips

2008-05-12T03:38:00.000-07:00

I'd like to share two tips of using mapping.It takes me along time to move lines from a page to another page in mapper.I was removing lines then added these lines to other pages.I have an easier way to do this task.
First Select the lines that you want to move

Then drag the selected lines and drag them to the grid of page2.

The second tip is that you can replace the functoid without removing lines related to that functoid. Just drag the new functiod on the old one.
in the example below I drag is not equal functoid on is equal functoid.
Then you will notice the lines were not removed.

In-Process Host vs Isolated Host

2008-05-12T03:22:00.001-07:00

The BizTalk Host object represents a logical set of zero or more runtime processes in which you can deploy services, pipelines, and other artifacts. The Host object also represents a collection of runtime instances (zero or more) where the deployed items physically run.

In-process Hosts:It represents what the administrator creates with BizTalk Server Administration Console like orchestration,receive and send port.On The other hand,Isolated Hosts reprensts what the BizTlak Developer creates at runtime like web services and WCF Services.

Recommendation:
It is recommended to build a host for send handelers , a host for receive handlers and a host for processes (orchestrations)
for more details check here

Best Practices for the Sql Adapter

2008-05-11T23:28:00.000-07:00

It is preffered to use sql adapter instead of using the C# code when you want to select,insert,update data in DB because microsoft follow the best performance ways to implement these task.
and here is the link of best practices for the sql adapter

Dana Trip

2007-10-30T01:27:00.000-07:00

Dana is one of amazing place in Joran.I have visited in April

JorDev Session

2007-08-21T02:40:00.000-07:00

I attend JorDev Session about Thinking Community

Suleiman Shakhtour Integration Tips

BizTalk Server: Receiving SAP IDOC Messages

Introduction

Problem

Solution

Conclusion

BizTalk Server: Extending WCF Adapter

Introduction

Problem

Solution

Conclusion

See Also

Dynamic Mapping Resolver

Introduction

Where do we use mapping?

Problem

Solution

What is dynamic map resolver?

How to implement dynamic map resolver?

BRE implementation

Dynamic map resolver in Inbound Maps

Dynamic map resolver in Outbound Maps

Dynamic map resolver in Orchestration

Sample Code

Conclusion

See Also

Creating Custom Adapter Provider in ESB Toolkit SFTP As an Example

Introduction

Problem

Solution

Test the solution

Sample Code

Conclusion

See Also

BizTalk Server:Using Global Variable in Mapping

Introduction

Problem

Solution

Sample Code

Conclusion

BizTalk Server Dynamic Schema Resolver Real-life Scenario

Introduction

Bad Solution

Adding New Source Steps

Adding New Destination Steps

Problem

Good Solution

Creating BRE Policy

Create Custom Pipeline Component

Create Custom Receive Pipeline

Create Custom Send Pipeline

Create Receive and Send Pipeline artifacts

Sample Source Code

Conclusion

See Also

How to Extract Email Attachments By Pipeline

Introduction

Problem Statement

Solution

Create Custom Receive Pipeline

Create Receive Pipeline

Configure POP3 Adapter

Test the Solution

Sample Code

Conclusion

See Also

An Easy Way to Get BizTalk Alerts

Introduction

Problem Statement

Solution

Conclusion

How to Configure SFTP Send Port Dynamically in BizTalk Server 2013

Introduction

Problem

Solution

Step By Step Implementation

Sample Code

Conclusion

See Also

How to use SFTP Adapter using BizTalk Server 2013