sfeldman.NET

BizTalk Custom Pipeline

Sean Feldman — Thu, 05 Nov 2009 14:17:00 GMT

I am starting a project that involves a lot BizTalk 2009. Since this is a completely new territory for myself, I decided to blog about it. There a lot of resources available out there, especially from MSDN.

First thing I decided to look at was a Custom pipeline. Two types of pipeline exist – sending and receiving. To understand better pipelines, I read BizTalk 2004: A Message Engine Overview on MSDN. It provides more information than just pipelines that is very useful for general BizTalk understanding (which I find a vital if you plan to develop for BizTalk).

A typical receive pipeline has 4 sections (or “component areas”) and send pipeline 3.

In BizTalk project these pipelines are presented as a flow, with designated areas where you can drop components designed especially for any given stage in a pipeline of a certain type. You can develop pipeline custom components in managed code, but you have to link it to the designer. How that is done? Each component has to be marked with a ComponentCategory attribute to indicate where in pipeline it can be assigned. A component that should be only assigned to Decoding section would be marked as:

     [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_Decoder)]

    [System.Runtime.InteropServices.Guid("9D0E4103-4CCE-4536-83FA-4A5040674AD6")]

The interoperability GUID 9D0E4103-4CCE-4536-83FA-4A5040674AD6 stands for “receive pipeline, decode section” and it is used to indicate to Visual Studio designer where the component can be dropped. For this purpose, I have create a component called “CustomPipelineComponent” and decorated with these attributes. Dragged and dropped the component and this is how the BizTalk pipeline looks like after that (XML view):

    <?xml version="1.0" encoding="utf-16"?>

    <Document xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" PolicyFilePath="BTSReceivePolicy.xml" MajorVersion="1" MinorVersion="0">

      <Description />

      <Stages>

        <Stage CategoryId="9d0e4103-4cce-4536-83fa-4a5040674ad6">

          <Components>

            <Component>

              <Name>SimplePipeline.CustomPipelineComponent</Name>

              <ComponentName>SIMPLE Pipeline component</ComponentName>

              <Description>CustomPipelineComponent</Description>

              <Version>1.0.0.0</Version>

              <CachedIsManaged>true</CachedIsManaged>

            </Component>

          </Components>

        </Stage>

        <Stage CategoryId="9d0e4105-4cce-4536-83fa-4a5040674ad6">

          <Components />

        </Stage>

        <Stage CategoryId="9d0e410d-4cce-4536-83fa-4a5040674ad6">

          <Components />

        </Stage>

        <Stage CategoryId="9d0e410e-4cce-4536-83fa-4a5040674ad6">

          <Components />

        </Stage>

      </Stages>

    </Document>

If you look close, each section in pipeline (or Stage) has a designated GUID. Decoder has 9D0E4103-4CCE-4536-83FA-4A5040674AD6, same as we used to decorate our component with.

Now the saucy part – how do I debug and test it?

To debug (which is NOT testing) I used BizTalk 2009 SDK utility called Pipeline.exe and located in %ProgramFiles%\Microsoft BizTalk Server 2009\SDK\Utilities\PipelineTools. One disadvantage I found with this utility is the fact that you have to deploy your custom pipeline component assembly to BizTalk 2009 designated location (%ProgramFiles%\Microsoft BizTalk Server 2009\Pipeline Components). Well, at least you can debug your custom component. For that I had to update the project settings of my custom pipeline component project and setup 3 things:

Specify Output for build artifact (Figure 1)
Set Start Action to Pipeline.exe (Figure 2)
Specify Command Line Arguments to use pipeline that contains custom component

Figure 1

Figure 2

When a new instance of Debugger is invoked on component project, debugger will kick in and stop at breakpoints.

To test a custom component you will have to dig dipper. I have figured a few things:

You cannot do unit testing. BizTalk has integration testing baked in.
You have to include BizTalk test assemblies, located in %ProgramFiles%\Microsoft BizTalk Server 2009\Developer Tools (I wish other blogs would mentions this folder!)
You will have to cast your objects to BizTalk base classes (the ones your maps/pipelines/schemas) in order to invoke testing methods (such as map.TestMap(…), schema.ValidateInstance(…), and pipeline.TestPipeline(…) accordingly)

I found one good blog post about how actually to write a state based testing, so I will not reproduce it here.

PS: an attempt to test pipeline components as unit testing is showed in this blog.

Lost Generation – Really Good

Sean Feldman — Wed, 28 Oct 2009 20:36:00 GMT

Creativity

Sean Feldman — Tue, 27 Oct 2009 03:02:00 GMT

One picture is worth a thousand words :)

Presentation: From Good to Great Developer

Sean Feldman — Fri, 23 Oct 2009 01:52:00 GMT

I really liked this presentation. Highly recommended.

LINQ to XML for Better Maintainability

Sean Feldman — Fri, 09 Oct 2009 03:28:00 GMT

Today I was trying to solve a simple technical problem. Given a specific XML, needed to clean it up by removing any elements of a particular type.

<Attachment>

  <Name>file1.pdf</Name>

  <Id>1</Id>

</Attachment>

<Attachment>

  <Name>file2.pdf</Name>

  <Id>2</Id>

</Attachment>

Result had to be without Id elements

<Attachment>

  <Name>file1.pdf</Name>

</Attachment>

<Attachment>

  <Name>file2.pdf</Name>

</Attachment>

A few choices for implementation:

Regex
XmlDocument
LINQ to XML
XSLT (as suggested in comments)

Regex option is probably the most efficient, but not the most maintainable. Myself, looking sometimes at the solutions with Regex I ask “what the heck did I try to do here”. So much for “code doesn’t lie”.

XmlDocument is more expressive than Regex option, but way too chatty.

LINQ to XML same as XmlDocument, expressive. As well as very clear and fluent. I picked this option not for performance, but for maintainability sake. I know it will take less developer type to understand and/or modify code when it’s time to change it. And it documents itself very well, with no need to write any comments.

var xdoc = XDocument.Load(new StringReader(received_content));

xdoc.Descendants().Where(element => element.Name == "Id").Remove();

return xdoc.ToString();

Note: this is a very specific case, which does not indicate it’s a solution to all kinds of problems. Regex / XmlDocument are valid tools for all sorts of other problems.

Too Loosely Coupled Code – One Year Later

Sean Feldman — Wed, 07 Oct 2009 02:00:00 GMT

More than a year ago, I have posted a blog entry related to what I was trying to implement in one of the projects. Unfortunately, not my team could understand what I was trying to do, neither I was able to make myself clear. Either way, I ended up closing the blog with a question “can code be too loosely (coupled), or your code is so coupled, that anything else is difficult to digest?”. Now I can answer my own question question.

Today, my current team had a team learning session, when the latest project demonstration showed what are we doing with a IoC container we started to leverage (StructureMap). The team is not the same one I had back a year+ ago. This team has also had same difficulty in the beginning – code that was too coupled. We have started from “Poor mans’ injection”, moved to home-grown Dependency Resolver with a typical Startup Task, and now have reached the point, where a third party open-source IoC container is used. I think that could be achieved because:

We worked on decoupling code, learning the pros and cons
We got to the point were SRP is followed
We have got to the point were Design by Contract is almost entirely how we develop
We have a complete TDD development in place

So why I could not root the idea in my previous team? Because it was way too early for my team mates and I was not ready to wait.

Oh, and how the code looks now? Even better than before ;)

public class ComponentB : IComponentB

  private IComponentA dependency;

  public ComponentB(IComponentA dependency)

    this.dependency = dependency;

  public void SomeFunctionality(string param)

    var result = dependency.OperationDefinedByDependencyContract(param);

    return DoSomethingWith(result);

Benefits:

TDD/BDD development style is enforced better when there’s no default constructor with Dependency Resolver or Poor’s man injection
No messy Startup Task that is a must when doing Dependency Resolver
Can leverage smarts of IoC container to perform tasks on a massive scale (interceptors, events, etc)
Easy way to verify if all dependencies are wired or not (contract based dependencies)

And as one commenter of the original blog post has asked correctly: “The question might be more appropriately be, is the code self-documenting? Can someone who did not write the original code follow the logic and modify and maintain it?”. The answer is: it depends.

You have to get to a certain level to answer Yes. Until you get to that level, the answer will always be No. In order for code to be self-documenting, it needs to be of high quality and accompanied with tests/specifications. In order to be modified, maintained, and logic followed by a person who did not write it, it has to have the listed above AND have the person to be at a certain level of skills.

Consuming ASMX Web Service With WCF

Sean Feldman — Fri, 02 Oct 2009 06:23:00 GMT

ASMX web services were a breakthrough when appeared on .NET platform. A lot of services were created to take advantage of web services technology.

Now, that WCF is replacing legacy web services, we still have a lot of legacy web services running and being used. Connecting to legacy web services from WCF can be achieved either by leveraging Visual Studio .NET auto-generated proxies, or by creating dynamic channel I talked about before. Creating dynamic channel looks like a cleaner solution, but there are a few caveats:

1. Legacy web service does not always expose a known/published interface (quiet often just implementation and no contracts);

2. Configuration might be a little tweaky;

In order to overcome the 1st issue, we can “re-invent” the contract that a legacy web service “would” implement.

     [XmlSerializerFormat]

    public interface ILegacyService

    {

      [OperationContract]

      Version GetVersion();

    }

Next we can generate a dynamic channel

    var channel = new ChannelFactory<ILegacyService>().CreateChannel();

    using (channel as IDisposable)

      return channel.GetVersion();

Once we have channel – time to configure

    <system.serviceModel>

      <client>

        <endpoint name="SomeServiceEndPoint"

                  address="http://some.where.com/LegacyService/LegacyService.asmx"

                  binding="customBinding"

                  contract="ILegacyService"

                  bindingConfiguration="custom.binding.for.SomeService">

        </endpoint>

      </client>

      <bindings>

        <customBinding>

          <binding name="custom.binding.for.SomeService">

            <textMessageEncoding messageVersion="Soap12" writeEncoding="utf-8" />

            <httpTransport/>

          </binding>

        </customBinding>

      </bindings>

    </system.serviceModel>

One comment about ServiceContract attribute. When consuming a legacy web service, and it has a pre-defined namespace. Modification to “re-invented” service will include the required information (such as Namespace)

    [XmlSerializerFormat]

      [ServiceContract(Namespace = http://some.where.com/LegacyService)]

      public interface ILegacyService 

      {

        [OperationContract]

        Version GetVersion();

      }

To Test or Not To Test: Not a Question!

Sean Feldman — Wed, 30 Sep 2009 03:55:00 GMT

There was a lot of noise generated by a blog “The Duct Tape Programmer”. There were lots of responses as well (link, link, link, link, link, link, link, link…). The last one I read was from Oren, where he puts it plain – just don’t do it.

I have my reasons to disagree with the statement(s) Joel wrote in his blog, but probably the most important one is about testing. Probably the reason for that is because I was developing without TDD for quiet a while, and I saw the difference. To me tests are design, and if you drop them, well, hell with design. A good project has to have a good design, and design starts from code, expressed first in tests, and after that in production code.

PS: if you disagree, the trash bin in on your desktop, not at my blog comments. 10x

Dynamic WCF Proxy

Sean Feldman — Wed, 23 Sep 2009 01:22:00 GMT

Our new system is entirely based of services (SOA solution). From the day one we had an issue with Visual Studio auto-magically generated proxies and management of those as system grew. Solution at that time was to create clients of the services dynamically, but the knowledge of WCF we had was a minimal. Now, 6+ months later, we finally getting to the point where I am comfortable and pleased with the solution. The interesting part is that WCF had that option all the time, we were not just educated enough to see it. Now we are.

The solution is to leverage ChannelFactory provided by WCF and create a client proxy from an end point defined in configuration file. Let me show the process from the client perspective:

     var channel = new ChannelFactory<IWcfAppenderService>("WcfAppenderServiceEP").CreateChannel();

channel is our proxy. Reading carefully Programming WCF Services book from Yuval Lowy (page 48), it is clear that a channel has to be closed, regardless of the state it’s found in after invocation (faulted or not).

    using(channel as IDisposable)

              channel.Append(loggingEventDataDto);

.NET using statement does the work.

This is it. To simplify the whole thing, we could intercept method calls on channel with a transparent proxy of ours and wrap with using statements – that way a user does not have to remember to do it each time.

Configuration file for a client would contain the minimal required information:

    <system.serviceModel>

      <client>

        <endpoint address="http://localhost:4268/WcfAppenderService.svc"

            binding="wsHttpBinding" 

            contract="WcfAppender.Contracts.IWcfAppenderService" 

            name="WcfAppenderServiceEP">

        </endpoint>

      </client>

    </system.serviceModel>

And the last missing part – contracts. A shared assembly with contracts would define all the service and data contracts. This is the only coupling between client and the server, which is logical, because contracts are coupling.

I am not done with my WCF adventures, but I can definitely point to a great book by Yuval Lowy as a reference for WCF.

Goodbye Launcy, Hello Executor

Sean Feldman — Tue, 22 Sep 2009 16:44:00 GMT

I have recently switched from Launchy to Executor after Terry told me about it. I test drove it for a while and find it better than Launcy, which was a great tool. Time for myself to move on to a better tool.

Pair-Programming Article

Sean Feldman — Mon, 21 Sep 2009 21:11:00 GMT

While reading blogs during today morning, I read an interesting one, which captures the way I feel about Pair-Programming. I have my opinion before (old posts), and it’s slightly updated since then, but at the end I still think it’s a great way and not just to develop.

TODOs are not forgotten

Sean Feldman — Thu, 17 Sep 2009 17:13:00 GMT

We are using Hudson as a build server, and one of the lasts steps that were taken is to mark a build as ‘unstable’ when we pass a certain number of TODO comments in our code (an arbitrary number). While I am not a 100% sold on a number, I think it’s a good way of insuring things are not just marked and forgotten. Actually, we are not even tracking TODOs, but BROKEN_WINDOW comments, as those are definitely bad. Failing on HACK is another possibility. Visualization plays a significant role in my case (interpretation of things based on visualization), and here how it looks (green is all good, yellow is all passed, but number of comments has exceeded the limit).

Gallio Warning Output

Sean Feldman — Tue, 15 Sep 2009 00:23:00 GMT

Today I was trying to figure out how to access output Gallio is using to render a warning message. According to this, the Assert.Warning was replaced by TestLog.Warning.WriteLine. But for some reason that is not doing the job. Has anyone encountered similar problem?

We run today into this problem, while trying to update our spec-based testing framework. We got the required details from with-in the base specification (test) leveraging TestContext.CurrentContext.LogWriter.Warnings, but with no success. Anyone, ideas?

NHibernate Profiler

Sean Feldman — Sat, 12 Sep 2009 06:30:00 GMT

Recently I had to update our domain model and it required some NHibernate profiling. I have installed trial version of NHibernate Profiler, and it rocked. The fact that it was not only able to show what was going on, but also give suggestions how to improve NHibernate usage just rocked (unlimited records returned).

I also loved straight, cut-the-bullshit communication with Oren, who has responded quickly (to the bug that I found). Worked for me, and we are going to purchase a few licenses.

Wix Product Version with Build Script

Sean Feldman — Wed, 02 Sep 2009 04:49:00 GMT

Our projects are deployed with MSIs. Each MSI file is appended with the product version number (regular staff – major, minor, build number, revision number).

While this is enough for deployment, in production, it is hard to determine what product version is installed without either searching for assemblies to see the version, or some other piece of evidence. Today I decided to leverage Wix to generate the Product Version number, based on our NAnt script.

Nant script has all parts of product version defined as follow:

  1: <property name="version.major" value="1" />
  2: <property name="version.minor" value="7" />
  3: <property name="build.number" value="0" readonly="false"/>
  4: <property name="svn.revision" value="0" readonly="false"/>
  5: 
  6: <property name="project.version.full" 
  7:           value="${version.major}.${version.minor}.${build.number}.${svn.revision}" 
  8:           dynamic="true" readonly="false" />

Next step was updating Wix installer project file to include an external include file with a defined variable for product version, called ProductVersion, and apply that to project. Here’s partial Wix installer file:

  1: <?xml version="1.0" encoding="UTF-8"?>
  2: <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi" xmlns:iis="http://schemas.microsoft.com/wix/IIsExtension">
  3: 
  4:   <?define build_dir = "$(var.ProjectDir)..\..\build\compile"?>
  5:   <?define documentation_dir = "$(var.ProjectDir)..\..\docs"?>
  6: 
  7:   <?include $(var.build_dir)\ProductVersion.wxi ?>
  8: 
  9:   <Product Id="5646ced5-d897-4978-b1d6-338e5baadbe9"
 10:        Name="Cortex Central Business System $(var.ProductVersion)"
 11:        Language="1033"
 12:        Version="$(var.ProductVersion)"
 13:        Manufacturer="Cortex Business Solutions"
 14:        UpgradeCode="790b1905-c843-4973-a078-79874a7f30f2">
 15:

Include file, ProductVersion.wxi is the one that should contain variable ProductVersion. This include is generated by nant script, injecting nant project.version.full variable.

Nant script to generate Wix include file:

<echo file="${build.compile.dir}\ProductVersion.wxi" message='&lt;?xml version="1.0" encoding="utf-8"?&gt;&#xA;&#xD;&lt;Include&gt;&#xA;&#xD;&lt;?define ProductVersion="${project.version.full}"?&gt;&#xA;&#xD;&lt;/Include&gt;' />

Which results in a file, that contains the required product version variable for Wix installer:

<?xml version="1.0" encoding="utf-8"?>
<Include>
  <?define ProductVersion="1.7.1234.4321"?>
</Include>

Once MSI is installed, Add/Remove programs contains the name with the version: