Yossi Dahan [BizTalk]

Another nice online SBUG meeting

2009-10-23T09:18:00.002+01:00

This time Global 360 will introduce themselves to the group members, if you’re curious – check the details here

Online SBUG meeting

2009-10-12T15:21:00.001+01:00

Check out tomorrow’s ONLINE SUBG meeting in which Jean-Pierre Auconie, the engineer behind the famous MessageBoxViewer tool talk us through it; details here

Don’t be scared of custom disassemblers

2009-09-27T10:53:00.001+01:00

During the last few weeks I’ve been asked to review two separate projects, for two separate companies, developed – naturally – by two separate teams.

The two things both projects had in common were that they both had to deal with legacy “flat files” and they both chose to process these files outside BizTalk using custom code.

In both these cases I completely agree with the decision to use custom code to parse the incoming files – as good as the out-of-the-box flat file support in BizTalk is (made significantly better with the introduction of the flat file wizard, once one gets the hang of using it) – there’s no avoiding from writing custom code to parse flat files every now and then – some file formats are pretty challenging with different records types, conditional records, interleaving segments etc.

I do not agree with, however, the decision to perform this custom parsing outside BizTalk.

I’m pretty sure I would not even bother posting this point had I not seen two of these in the same month, but the fact that I did suggests it may be worth posting a quite note.

One of the projects had the code in a console app, called from a windows scheduled task; the application would pick up files from a folder, parse them, and drop the xml representation in another folder, for BizTalk to consume.

The other had a windows service monitoring a folder and pick up any files, parse them to a different, simplified, flat file format (!), and drop them in another folder for BizTalk to consume.

Both of these introduce another component to the mix; such component needs it’s own error handling, it’s own monitoring, deployment strategy, operations manual etc. similarly it includes a fair bit of re-inventing the wheel – writing code to monitor folders, read files, and write files – stuff that BizTalk is doing out of the box.

What would have been the correct approach then? quite simple – a custom disassembler in the receive pipeline -

Writing a custom disassembler it quite simple - at the end of the day, it boils down to developing a class library, which implements a few simple interfaces, the main one – IDisassemblerComponent defines two methods - Disassmeble and GetNext (the other interface are even simpler, almost insignificant in terms of effort)

Disassemble gets the source message and potentially parses it up front, GetNext is called repeatedly by the pipeline to receive 0 or more parsed messages, until it returns null. simple.

In one of the projects I’ve since taken their existing code (console app), refactored it into a class library, and wrapped it in a custom disassembler class that calls it; converting the scenario to a BizTalk pipeline and performing the key “developer testing” took less than a day.

Why did they not do it to begin with? whilst sometimes there are valid reasons, technical or otherwise, I suspect that in this case it was just unfamiliarity with with BizTalk and some lack of confidence in the development team’s ability to learn and implement (or their belief in themselves); these are valid concerns to any project manager, but I would suggest that a better course of action would have been to spend some time looking at what it takes to implement a custom disassembler, seeing that’s its not at all that scary, and by doing so learning more about a product used in the solution (BizTalk) and achieving a better architecture, and more maintainable approach.

Paolo Salvatori on ways to create messages in orchestration helper classes

2009-09-13T11:23:00.001+01:00

I’ve been toying with message creation a few times in the past, and recently turned to Paolo for help with a question; Paolo has an amazing blog and he has now posted posted some of his wisdom around ways to create messages from a helper class to an orchestration on his blog; well worth reading (as any entry on this fantastic blob really!)

Loosely Coupled….Part II

2009-09-11T14:33:00.001+01:00

In a previous post I’ve mentioned our constant attempt to strike the right balance when it comes to loosely coupled services; I’ve mentioned that we were looking at two different scenarios – loosely coupling calls to services outside BizTalk and loosely coupling calls to services inside BizTalk (once implemented within the BizTalk group)

I’ve also mentioned that our solution is composed of a few distinct ‘areas’ (each one generally encapsulated as a BizTalk Application), which we consider, in most cases, to be service boundaries, and – within one ‘flow’ of an incoming request message, we will often have to cross one or more of these boundaries to achieve our end goal.

In most cases, in our solution, the ‘subscriber’ service would use the schema of the ‘publisher’ service for its incoming message; this roughly follows the principle of a service’s proxy, albeit a bit upside down - for practical reasons. Only that - and that’s a much bigger difference - we don’t create a copy of the schema as a service proxy would, but rather reference the schema of the publisher directly (through a shared assembly); this, of course, creates a strong dependency between the two and - over time – this has caused us a lot of headache around deployments as whenever we wanted to update the publisher, we’d have to remove the subscriber too.

Recently we have experimented with following more closely the service proxy approach and instead of using the same referenced schema (using a shared assembly), we’ve created an identical copy of it – same root node and namespace - in the ‘subscriber’ side.

The assumption was that we will be publishing a message using the copy of the schema the publisher holds and be receiving it using the copy of the schema the subscriber holds, but as the message itself will look exactly the same, and will inevitably have exactly the same message type, and so it would be picked up by the subscriber successfully.

Had it worked, we would be have been able to avoid the dependency between the subscriber and the publisher, which would help us gain much needed flexibility in the publisher to support, and change for, multiple subscribers.

Theoretically - if the publisher schema had to change (say – to support functionality required by other subscribers), as long as the change is backwards compatible such as added elements, we could replace the publisher copy of the schema, but leave the subscriber copy as is, until such a point that we need/want to update the subscriber process.

Well - in BizTalk 2006 – this would have worked just fine; unfortunately – from R2 onward, it no longer works – when an orchestration receives a message, it often does so based on a subscription that included the ‘messaging message-type’ (root node and namespace); however – starting with R2 – an additional check has been introduced – to compare the full .net type name of the schema used by the publisher message with the full .net type name of the schema used by the subscriber, assembly, version and all.

This check obviously fails in our scenario, and our fancy loosely coupling solution no longer works (in R2 or 2009).

I think this check is actually a result of code introduced as a hotfix for BizTalk 2004, which – for on reason or another did not make its way into BizTalk 2006 but did into later versions, but I’m not sure, either way – it’s important to note the workaround described at the bottom of the hotfix description, as it appears this behaviour can be turned off, but one would have to check carefully the potential impact.

What else could we do?

Well – one pattern we know that works fairly well is the broker pattern – there’s the publisher, with it’s own schema, there’s the subscriber, with its own – completely different schema, and there’s a broker – a third process that has dependencies on both and contains a map to convert one to the other; on the plus side – this gives us all the flexibility we need – at any one point we only need to deploy two entities – the publisher or the subscriber and the broker, which is good enough; having the process, with a map, allows us to use multi-part messages if we deem them suitable, and all the complexity we need in the mapping; on the down side there are more artifacts to deploy and manage but, more crucially, one extra message box hop which, in a low latency scenario as ours, is not a small price to pay.

Another option is to simply expose the subscriber as a service and call it as such – there are big benefits to doing that – including the fact that we can now have a copy of the schema, in the form of a proxy or without one, and we have also decoupled the services in terms of BizTalk groups –the other service can be anywhere, although this was never a requirements for us; however – we’re paying in more pub/sub again, as well as more IO and quite possibly more complexity.

Theoretically we could have also use XmlDocument (or any other generic wrapper, for that matter) to convey the message, but a) I don’t like typeless intechanges and, b) this does not work well in cases where correlation is required, as the following receive tend to short-cut the subscriber and pick up the request as a response, that is unless you’re willing to introduce two wrappers – one for the request and another one for the response.

Debugging Expression shape code

2009-09-08T08:20:00.001+01:00

Sandro Pereira has posted a question, and answer, in the BizTalk newsgroup (he also described his answer, in detail, on his blog) about debugging expression code in Visual Studio

He wasn’t referring to debugging code in helper classes, but code in expression and assignment shapes.

My answer was that this was not possible, but Sandro quickly proved me wrong, as he demonstrates in his answer and blog post, and this got me thinking –firstly – despite knowing about the option to use the generated code (and actually using it on very rare cases to understand a certain BizTalk behaviour) I had never thought of using it for debugging, and that is an interesting thought.

However – I had to wonder – how come I never came across the need to? in all those years of BizTalk development, not even once can I remember thinking – oh! I could solve this if only I could debug the piece of code in this shape..

The reason, I think, is two fold -

1. I rarely have more than 2-3 lines of code in an expression shape of any kind; if it’s not straight assignments, its going into a helper method; it’s cleaner, it’s more reusable, and it’s easier to debug.

2. I use trace. a lot. and so every few shapes or so, and certainly in expression shapes, I will have a trace line that outputs to a log file important information about the state, and the flow of the process; this proves to be invaluable when troubleshooting issues on the live environments, but also really helpful in development.

On ASMX, WCF, namespaces and generated schemas (in BizTalk 2006)

2009-09-07T22:18:00.001+01:00

Recently we’ve started to consume a new version of a web service we’ve been using for a while.
We’ve known that, as a whole, not much had changed, only that they have now moved to WCF; they would have migrated their classes to VS 2008 but would expose pretty much the same functions, using pretty much the same parameters.

Still – it appears that BizTalk now insists on generating multiple schemas for the web reference, and as more of the service is moved across more schemas are introduced.

This caused Oleg a fair amount of pain as, when new schemas would be introduced, they would re-order the existing schemas, so reference1.xsd (in the web reference) would suddenly become reference2.xsd, which in turn break out maps.

The process of finding out the logic behind which schemas are created was fairly short and simple, but as I’ve documented it I thought I might as well share it -

Initial observation revealed that whilst the ASMX services’ WSDL file contains all the schemas needed, the WCF services using import statements in the WSDL file; the schemas exist in separate ‘files’.

The ASMX services always uses the XmlSerializer, WCF services use the DataContractSerializer by default, but can be configured to use the XmlSerializer if required.

Here’s a walk thorugh of the scenarios we’ve compared (using BizTalk 2006) -

Standard WCF project, DataContractFormat
We’ll start by comparing the standard WCF sample project generated when you create a new WCF service application in Visual Studio 2008 -

[ServiceContract]
public interface IService1
{
    [OperationContract]
    string GetData(int value);

    [OperationContract]
    CompositeType GetDataUsingDataContract(CompositeType composite);
}

[DataContract]
public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Looking at the WSDL generated, 3 schemas are imported –

1.       The usual generic types

2.       The definition of the compositeType type

3.       The definition of the service’s messages (GetData, GetDataResposne, GetDataUusingDataContarct, GetDataUsing DataContractResponse)

Adding a web reference to this service from a BizTalk 2006 project we can see it represents this fairly accurately -

We can see all 3 schemas downloaded from the service, but within the reference.map generated code a single reference.odx defined the methods in the form of ports and web-messages, and reference.xsd defineds the compositeType schema.

Equivalent project in an ASMX service

I’ve created an equivalent ASMX service, which looks like this –

[WebService(Namespace = "http://tempuri.org/")]
public class Service1 : System.Web.Services.WebService
{

    [WebMethod]
    public CompositeType HelloWorld(CompositeType composite)
    {
        CompositeType response = new CompositeType();
        response.StringValue = "Hello World";
        return response;
    }
}

public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Publishing this service I can see it’s WSDL contains (does not use import, but that proved to be insignificant) a single schema that represents the service’s messages and the compositeType definition.

Consume this service from a BizTalk 2006 project and only the WSDL file is downloaded (there are no ‘external schemas’ to worry about) but within the reference.map pretty much the same odx and xsd files are generated, no real difference between ASMX and WCF here.

Next I’ve looked at changing the serializer the WCF service works with from DataContract to XmlSerializer –

Standard WCF project, XmlSerializerFormatter

Now we will change the serializer to XmlSerializer by adding XmlSerializerFormatAttributre to both the service and the data contracts

   [ServiceContract]
    [XmlSerializerFormat]
    public interface IService1
    {

…and

[DataContract]
  [XmlSerializerFormat]
  public class CompositeType
  {

The WSDL in this case includes only one import, for a single schema representing both the service messages and the compositeType schema (basic types are not exposed) and BizTalk now only has one schema downloaded, but again – the reference.map code remained identical (one ODX, one schema)

How will adding a second namespace affect these behaviours? Lets investigate -

WCF project, two namespaces DataContractFormat

To demonstrate this I’ll add another data contract - AnotherCompositeType, specify an explicit namespace for it and include it as a second parameter to the GetDataUsingDataContract operation -

[DataContract(Namespace="HttpL://SomeNamespace")]
    public class AnotherCompositeType
    {
        bool boolValue = true;
        string stringValue = "Hello ";
 
        [DataMember]
        public bool BoolValue
        {
            get { return boolValue; }
            set { boolValue = value; }
        }
 
        [DataMember]
        public string StringValue
        {
            get { return stringValue; }
            set { stringValue = value; }
        }
    }
 
        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, AnotherCompositeType anotherComposite);

Using DataContractFormat again, but with two classes, representing two different namespaces, we’re now getting yet another schema - the fourth one - representing the added data contract (if the namespaces of both data contracts were the same, the DataContractFormat would have included them in the same schema)

On the BizTalk side, the reference.map code now also contains a second schema, one describes the original CompositeType, and a second describes the second type – AnotherCompositeType and here as well – were the two types in the same namespace, a single schema would exist, describing both.

Let’s look at the same again, using the XmlSerializerFormat

WCF project, two namespaces XmlSerializerFormat

Adding the XmlSerializerFormat, I also have to remember to include the XmlRoot attribute to set the namespace, as the serializer does not look at the DataContract attribute -

[DataContract(Namespace = "http://SomeNamesapce")]
[XmlSerializerFormat]
[XmlRoot(Namespace = "http://SomeNamesapce")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Now the WSDL for this service, using the XmlSerialiserFormat imports two schemas – one for the service messages, and the CompositeType schema, all reside in the same namespace, and a second for AnotherCompositeType which is defined in a separate namespace

Consuming this from BizTalk and again I’m getting two schemas – one for each namespace.

So far – switching between DataContractFormat and XmlSerializerFormat made no difference to the generated code under reference.map, but it did change the way the WSDL is constructed (import vs. embededed schemas) and therefore the downloaded components (wsdl and schemas, vs. wsdl only)

Note - another thing I’ve noticed is that when a new schema needs to be generated under the reference.map code, as a result of a change to the service, updating web reference does not seem to do so; I had to delete the web reference and re-add it to see the newly added schema.

Last - let’s look at how the ASMX service behaves with two namespaces –

ASMX service with two namespaces

I’ve added the second class, and added it as a parameter to my web method

[XmlRoot(Namespace = "http://AontherNamespace")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

And, still consistent – when consumed from BizTalk 2006 I’m getting only the WSDL downloaded (two schemas are embedded) but the reference.map code contains two schemas – one for each namespace.

To summarise -

Using the DataContractFormat you will always get one schema for generic types, one schema for the service’s messages, and then one schema for each namespace any other types are declared in (0..n)

Using the XmlSerializerFormat schemas are embedded in the WSDL file, and you would get one per xml namespace used.

As far as BizTalk generated code is concerned, however, there’s no difference between the two.

What this meant to us – well – we understand better, but there’s still not much we can do.

In our case – we control the service, and – in fact - we know that the only reason we encounter multiple xml namespaces in the service contract is because the various classes exist in several .net namespaces, and they have not supplied the DataContract attribute on any class, they have certainly no supplied the namespace parameter to that attribute, which meant the .net namespace was used as the xml namespace, resulting with multiple namespaces and therefore multiple schemas.

One that team added the attribute, and used a consistent xml namespace throughout, our immediate problem was solved; however - had it been a third party’s service we would not have that luxury and we would have had to update our code whenever we update our web reference, even if only new types were added in a backwards compatible way (as the schema ordering may have changed)

On that – it’s probably easier to simply rename the schemas (and the underlying .net types) under the reference.map generated code rather than the referencing maps and messages.

Pro Business Activity Monitoring in BizTalk Server 2009

2009-09-03T07:42:00.002+01:00

Richard Seroter wrote a review of this book on his blog

Whilst I haven’t yet finished reading the book, I completely agree with Richard – this book is very well written, and is doing a fantastic job explaining this fascinating, and often misunderstood, if not completely overlooked, capability of BizTalk.

The book also does a very good job at looking at scenarios outside BIzTalk server, making it well worth reading for any enterprise solution architect.

Highly recommended.

Creating messages from scratch - revisited briefly

2009-09-02T13:39:00.002+01:00

A while back I’ve posted about the different ways to create messages in an orchestration, and later some performance comparison between them.

Mostly for fun I run a quick test on my newly installed laptop; I did not put nearly as much effort as I have previously, so don’t make out of these numbers too much, but I was amazed to see that all the results were running pretty much 10 times faster.

Now – it’s a new BizTalk (2009), new SQL server (2008), new operating system (Windows 7) and a new(-ish) laptop (Thinkpad T61), so there’s no way to know how much each component contributed to the improvement, but it is amazing how much difference can exist after just one year!

Well – not at all scientific, but I found it interesting anyway!

Webcast: BizTalk Server 2009 Performance on Hyper-V and Physical Deployments

2009-09-01T08:20:00.001+01:00

Ewan Fairweather is doing a web cast TODAY on BizTalk 2009 performance tests he’s done both on and off Hyper-V infrastructure.
I’ve seen bits of it before and is highly recommended- you can pretty much count on this being extremely useful if you’re serious with you BizTalk deployment.

Loosely coupled - Part I

2009-08-20T21:51:00.001+01:00

I’m working on a fairly large BizTalk implementation, which in many ways does not fit the classic BizTalk project pattern in that, whilst it has a large element of integration to it, it is even more a business process rich application.

One of the things that constantly keeps us busy is trying to strike the right balance between complexity/performance and maintenance, and a big part of that is finding just how loosely coupled the solution can and should be; this is something I find ourselves spending more and more time on.

There are two parts to this challenge for us – one is about ensuring our processes are not too tightly coupled with any of the many services we’re calling (internal or external) and the other is that different ‘areas’ of our solution, although implemented on BizTalk, and even in most cases within the same group, are treated as independent ‘services’ and are decoupled from each other; we’ve learnt the hard way the price of not getting good enough – mostly in terms of maintenance complexity and cost and are keen to find the right answers, if there are any.

In this part I would like to start tackling the first point – calling services -

In a recent post I ranted somewhat about the WCF’s adapter lack of support for multi-part messages, a topic I’m likely to come back to every now and then I suspect.

This was triggered by my attempt to prototype a way to call services from orchestrations in a loosely coupled way – ideally without the orchestration having to know about the service implementation (isn’t that the promise behind BizTalk Server?)

Naively, I started with a very simple plan – the orchestration would have its own representation for the service call, which it would publish to the message box; the send port, configured with the WCF adapter, would pick up the request based on some subscription, a map in the send port would convert the request to the service’s format and the service would be called; the response would be handled in a similar way.

I have documented that approach here

This worked pretty well in my little prototype, and I was rather pleased with myself for a short while, at least until Ben was tasked with taking this approach into our real world; it took very little time for it to break – all he had to do is call a real service - one that takes more than one parameter (how’s that for a lesson to not over simplify prototypes…)

To be fair, this needn’t be an issue, had we not used multipart messages in our processes, but – accustomed to using the SOAP adapter were using multi-part messages to represent the numerous parameters to the service, and – as I’ve hinted in my previous post – I do sincerely believe this to be the better approach.

That meant two things – 1. the WCF adapter would not be able to transmit the message, as it does not support multipart messages, and - 2. we would only be able to map the body part of the message.

These pretty much put a lid on my idea.

The WCF adapter’s lack of support for multi-part messages we could probably work around using a custom assembler or encoder – such a component could take a multipart message and convert it to a single part message using some pre-defined rules and/or part names, context properties, etc. but the map limitation really puts the lid on this approach.

What can we do then? well – we’d have to create a schema in our process that contains all the parts needed for the service calls, this does not have to look like the service contract, but it does have to contain all the information required to construct the service’s request and effectively means describing the multipart message in XSD; this makes me slightly uncomfortable as, in a sense, this schema exists specifically for this service; it also means we would have to work harder in the process to construct this message – on top of creating and deploying another schema, we would also need some map to convert the multiple messages containing the information to the single entity (or use some code to achieve the same), but it would work – once we have the single part message, we could publish it, map it to the service’s format in the port and deliver it to the service, in the exact same way that worked in my prototype.

At the moment, this looks to me as our only bet, but I’m looking for alternatives, if anyone has any suggestions?

A transform shape gotcha Update

2009-08-14T09:01:00.002+01:00

A while ago I blogged about this problem we've found with the transform shape; what I should have done, and haven't (I believe) is blog about the fix Microsoft issued as a result of our bug report.

You can read the details here

Interestingly I think the scope of the problem was bigger than we originally thought, certainly bigger that what I blogged about anyway, in that even if you don't have the tranform shape problem (for exmample - if you've created the orchestration on a machine with the hotfix, but then opened it in a machine without it) you can have some issues -

In one case we had a map, where the input was composed of several schemas; as you probably know BizTalk effectively creates a schema that aggregate all the parts into a single schema; to achieve this it had to impot several schemas from our solution, all share the same namespace;

When we tried to build the project with the map, we received errors saying a certain element type could not be found, the cause was that BizTalk was looking for it in the wrong schema, as it had several schemas imported into the same namespace, it chose the wonrg one and could not find the type definition.

Once we've installed the fix, and with no other changes, the map compiled all right.

BizTalk WCF adapter and multipart messages

2009-08-13T14:07:00.000+01:00

I wrote myself notes for this post months ago, but haven’t taken the time to write it down properly; a passionate discussion today with Rupert Benbrook prompted me to get back to this.

The WCF adapter does not support multi-part messages, and some would argue that’s a suggestion of the feature possibly being dropped in some future version of the product.

This small fact seems to have been glossed over by most – I, for one, haven’t heard this discussed anywhere, to the point that I’m left wondering if I’m really just “stuck in my ways”, but I honestly think that’s a poor decision.

Trying to search for some detail about this fact I found the following, rather old, but still very valid and informative, post; here are some of the key points made -

The reason support for multi-part message in WCF adapter is not being recommended is because WCF does not support it and the model proposed is that whoever wants to use something equivalent, need to aggregate the messages into one xml and send it as one single-part message.

(Tapas Nayak, Principal SDE, BizTalk Adapter Pack)

Multipart messages are a marginalized feature in BizTalk, the pipeline components like the flat-file and xml don’t understand them, same story for the majority of the adapters. So we decided to not support them in the WCF Adapter too

(John Taylor, Senior Technical Lead, BizTalk Server)

With regards to the first statement – I think that is somewhat down to interpretation – WCF did not change how web services work; WSDL is the same WSDL, specifications have not really changed; so – WCF is just a new (and of course – much better) implementation; this is an important basis for any discussion because it indicates that the WCF adapter could have taken the same path the SOAP adapter did.

Consider the following Service definition in WCF:

        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut);

or the equivalent web method in asmx:

        [WebMethod]
        public CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut)
        {
        }

the WSDL for both methods is pretty much identical, the operation definition looks as follows -

<wsdl:operation name="GetDataUsingDataContract">

<wsdl:input wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContract" message="tns:IService1_GetDataUsingDataContract_InputMessage"/>

<wsdl:output wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContractResponse" message="tns:IService1_GetDataUsingDataContract_OutputMessage"/>

</wsdl:operation>

the messages are defined as so -

<wsdl:message name="IService1_GetDataUsingDataContract_InputMessage">

<wsdl:part name="parameters" element="tns:GetDataUsingDataContract"/>

</wsdl:message>

<wsdl:message name="IService1_GetDataUsingDataContract_OutputMessage">

<wsdl:part name="parameters" element="tns:GetDataUsingDataContractResponse"/>

</wsdl:message>

And the schemas, in both cases, define the aggregated messages – with all the in/out parameters as required.

So – what is the difference? the way I see it is it is in the way the different adapters handle the messages as they pass through the adapter; the SOAP adapter has a layer in which, when receiving messages from the outside, breaks them to individual parts, delivering a multi-part message to the message box; similarly, when delivering messages from BizTalk to the service, it composes them, aggregating the various parts in the request message, to a single SOAP envelope; the WCF adapter does not include such functionality – it will take the message BizTalk provides and will wrap it with the SOAP envelope pretty much as is.

All my grievances about the SOAP adapter aside – I think its approach is a good one; it allowed the process to focus about the flow of the business process and avoided the need to define and construct artificial entities in the form of aggregated schemas, which often make very little sense (as the various parts can be largely unrelated from a process perspective, but still necessary for the service to perform its operation).

In my discussion with Rupert the same point was raised – the fact that SOAP supports one message in and one message out (i.e. no multipart messages), but in my view that’s a problematic argument – again – the whole point of the BizTalk architecture is that the send port can mask the transport mechanisms; ok – the SOAP adapter is not doing a particularly good job at that, but that’s hardly an excuse to make matters worse – I don’t want to have to know that I need an aggregated message to call this particular message because it uses the WCF adapter;

Moving on - I particularly disagree with the second statement mentioned - I think multipart messages are a very important feature in BizTalk, and would be surprised if any seasoned BizTalk developer would tell me they prefer not to use them (do let me know if you think I’m wrong); to start with – it allows one to define a type once and then use it multiple times; this makes maintenance infinitely better when types need to be changed (I thought it ought to be this schema, but found that this other schema is better); ok – so you have to be careful about how and where you re-use, to not introduce unwanted dependencies, but if you have large orchestrations which follow several steps, this is very useful.

Secondly, it allows one to easily link pieces of information that are only ‘somewhat-related’, especially important, in my view, when using pub/sub; having to pass several pieces of information from the publisher to the subscribers; the alternative, not surprisingly, would be exactly what is suggested above – create a schema defining the various pieces of information required and use that instead, but I think that’s a much larger maintenance overhead than a multi-part message definition; it is much easier to change a message definition that it is to change a schema; especially in large implementation with many different assemblies;

This was the point most passionately discussed with Rupert today – he argued that if a message is important enough to be published (with all its parts) it is important enough to create a schema for it; the point I was making, possibly not good enough, is that – to me, specifically in this area, the two are the same – I can define my aggregated message in a schema, that would have, say – 2 elements, each using a type defined in my other, existing schemas, or I could use a multi-part message, with two parts, each pointing at existing schemas; there’s very little difference in principal – I’m defining something – strongly typed and well defined, that is composed of two other, well known things, and I’m passing it around.

Obviously interoperability can only be achieved using standards like XSD, but that’s not a consideration at this point I’m publishing a message from BizTalk, which would be subscribed to by BizTalk, what’s wrong with describing it using BizTalk terms where conceptually there’s virtually no difference?

If there is no difference – why do I care? because multi-part messages are easier to create than aggregated schemas, and there’s less deployed artifact; I can simply put an assign shape and assign existing messages to the relevant parts; if I was to go down the aggregated schema route I would have to define the schema and create a map that would aggregate the messages; I could use code, or I could use xpath to assign the parts to the various elements, but I’d have to use a map or something to create the shell of the message first; seems like a lot of overhead for simply passing a couple of messages between processes.

I agree that many adapters don’t support multipart messages (the file and ftp adapters are the most prominent examples, outside the WCF adapter), and in many cases this makes perfect sense, but some do – SMTP and POP adapters, for example, use it very well; so does the SOAP adapter, of course.

There was one piece where we were in mostly in consensus, about the benefits of a single schema versus multi-part messages – when you wish to have a map in the port, I believe BizTalk would only consider the body part of the message, but again – to me that’s a shortfall in the product, not a proof that the existing feature is not useful; for both this and the pub/sub scenario I would have lover to be able to treat a multi-part message type in the same way a single-part message is treated – with a specific type that you can subscribe on (I can always set the message type manually, I guess, but would have been great to get out of the box support)

Similarly I agree that the built in xml and FF disassemblers don’t support multipart messages ( only in the sense that they will not disassemble them), but the model does – you can easily access, and parse, multipart messages from within a pipeline component, even a custom disassembler, so I can’t help wondering if the answer is in the following statement made by John Taylor, as quoted in that blog post -

The feature set for the WCF Adapter was potentially very large indeed and we were under some pressure to make cuts where we could.

and I think it’s a big shame.

I, for one, would be very disappointed if multi-part messages stop being supported, and I’m slightly worried – WCF is obviously taking more and more front stage, generally and with BizTalk specifically (even more so with the LOB adapter SDK and the BizTalk adapter pack that makes use of it), and as it does not support multi-part messages, it may well set the tone for future BizTalk developments.

Parallel shape behaviour in BizTalk 2004 and >2006

2009-07-26T08:46:00.001+01:00

‘Shiri’ had posted this question in the newsgroup -…

…After open the orchestration
debugger both at BT 2004 and BT 2006 we've recognized a different behaviour:
at 2004 executes through all the branches first shape and then back to the
first branch at parallel and executes the second ahape, the second branch
and etc. at 2006 executes the first branch - first and second shape, then
goes to the second branch and executes the first and second and etc.
It seems that 2004 works more like multithreaded then 2006….

I vaguely remember reading/discussing this difference in the past, but – unfortunately - I can't remember the exact details; I've floated this question around again, and I believe (but treat this with care – this may well be at least somewhat inaccurate) that there has indeed been a change in the behaviour of the parallel shape in BizTalk 2006, here’s some background -

The parallel shape was never intended to provide ‘true’ parallelism, and Microsoft has been fairly clear from the start that BizTalk will not process each branch on it’s own thread (which would have been required).

Darren Jefford explains very well the intention, and the expected behaviour, of the parallel shape in his book Professional BizTalk 2006, which I highly recommend; if you want a quick peek you can read the relevant piece here, the key point he makes is that when thinking about the parallel shape, you need to wear your business analyst hat, and not the developer hat – the parallel shape effectively “says” – hey - I’ll run this code (=branch), but if I reach a point where I’m sitting idle waiting for something to happen (receive, delay or listen shapes) I will go and run that other code (=branch) in the meantime; when I’m through with that (or reached a waiting point again) I will check if I’m ready to process the rest of the first bit of code…and so on…

This is not quite your techie run-things-in-parallel-on-multiple-threads approach, but – from my experience – it is more than enough (if you wanted to run things completely in parallel you could use BizTalk’s pub/sub, which would allow you to potentially get a lot more than just one thread – you might end up on a different machine altogether, for a price :-))

So – indeed – the BizTalk 2006’s (and subsequent versions) parallel shape behaves exactly as I understand it should do (and as is described in Darren’s book) and is consistent with Shiri’s observation - if you have three branches, and neither have a blocking shape – the left most branch will be executed completely, then the next one to its right and then the right most branch; however –and that’s a very important point to remember - Microsoft does not, to the best of my knowledge, guarantee any order of processing between the branches, and this might change in future versions (as indeed it has been seen from 2004 to 2006), so all you can assume is that all the branches can theoretically run in any arbitrary order or indeed in parallel.

Back to the 2006 behaviour - if, however, you had a receive shape as the second shape in the left most branch, when BizTalk would hit this shape it would move on to execute the second branch while it’s waiting for the message to be received; it would come back to the first branch at the earliest point once two things had happened – 1. the message it was waiting for was delivered and 2. it had reached a point in the currently executing branch in which it can stop and re-enter the first branch; this would be a receive shape, delay shape, listen shape or the end of the branch.

So – if that’s the 2006 behaviour, was the behaviour in 2004 different? yes, I believe it was – in 2004 the engine was, some would say, trying to be too clever - BizTalk 2004 would try to run branches on multiple thread if it can; where “if it can” depends on several factors, not the least the state of the thread pool at the time of evaluation; if it managed to do so, you would get code running truly in parallel, as Shiri observed, but there are no guarantees that this would be the case; in that sense BizTalk 2004 is less predictable than later versions of BizTalk, which is exactly the problem with this approach, and - considering that this was never the intention to begin with – I can fully understand the decision to simplify the model in BizTalk 2006.

Integrating DotNetNuke and the Geneva Framework

2009-07-22T09:07:00.001+01:00

Was a somewhat painful exercise. sure – it was not helped by the fact that I’m not really familiar with DNN, nor am I really a web developer by any stretch of the imagination, but never mind that – DNN has certain ‘features’ that made making it play nice with the WIF somewhat ‘challenging’. below are some points we’ve encountered that are worth remembering/considering (in no particular order) -

DNN will redirect to ErrorDisplay.aspx on most errors.
This means that once you’ve configured the web site with FAM and automatic passive redirects, most errors will send you in an endless redirect loop.
In the first few attempts I did to integrate the two, the aliases for my portals in the DNN database were only configured for ‘localhost’; as my STS is on a remote machine (very important for testing federated identity scenarios, obviously) I was now accessing my portal using the machine’s name and/or IP address, for which aliases had to be defined; this error was caught by DNN before the FAM had a chance to execute and so the call to ErrorDisplay.aspx was made when the user was still unauthenticated, but now it no longer carried a security token, which caused the redirect back to the STS and the infinite loop; to avoid that problem, I’ve added a location configuration in the web.config for ErrorDisplay.aspx and set it’s authorisation settings to allow all users – this allowed the error to be displayed despite of the fact the user is not authenticated (something that needs to be considered carefully, of course, but we’re not showing any dangerous details on our error page.)

The membership module performs a lookup for the username in DNN’s membership database.
As we’ve moved the authentication work from DNN to the STS, it is now possible for an authenticated user to not exist in the DNN database; we already have synchronisation process that keeps the two databases (our membership database, used by the STS, and the DNN membership database) aligned as we needed that anyway, but there’s always the chance of things getting out of sync; out of the box, the DNN membership module redirects the user back to the home page, in our case - because we’re using the FAM with auto passive redirects, this will enter an infinite redirect again (as the redirect to the homepage loses the security token), so we’re looking to change the on-error redirect to an allowed page (not an easy task in DNN, it appears)

DNN can host several portals.
DNN supports hosting many portals on the one application (/virtual directory) - driven by database configuration; that means that the FAM configuration, driven via web.config, does not fit very well as we’d have a single realm/replyTo address.
There were two optinos to choose from – we could decide that the entire DNN instance is a single RP, which would mean the existing configuration solution could suffice, but is a security risk – a user’s permissions cannot be checked at the STS at a portal level, only at ‘DNN level’; the other option was to treat each portal independatly, for this to work we had to set the realm and replyto values in the request to the STS dynamic as the configuration story was not enough; and so - we’ve extended the FAM by overriding OnRedirectingToIdentityProvider and setting the realm and reply properties of the SignInRequestMessage dynamically (based on the HttpRequest, in our case)

AudienceURI’s

Setting the realm and replyTo dynamically, as described above, raised another challenge - the Geneva Frameworks would like you to specify the audienceUris you’d be expecting in the tokens received from the STS; generally – this setting exists in the web.config, but as our realm can be one of many things we were faced with two options – either list all possible audienceURIs in the ‘allowed’ list, which has some security implications, or provide a mechanism to dynamically evaluate the request arriving and see if its allowed.

The problem with the first approach, out-of-the-box, is that it means keep updating the web.config of DNN with newly added portals’ Uris; this actually has two implications – one: it means that whoever sets up a new portal (which is a DNN user, generally), must have access to the config file – not quite what we had planned (or can live with), two: whenever the web.config gets edited, if we did allow that, the appdomain is reloaded, which kicks users our of their sessions (or so I’m told) as well as makes the next call really slow as the application is recompiled; the outcome was clear – we must stay away from the web.config

It appears that there are several ways around this:

You can have a single AudienceURI in the RP side, and have code your STS to always return the same one (for all portals, despite the realm provided); you will need a way to find the audienceUri to use (as its no longer the realm from the request, which is generally used), but that’s possible through configuration; you are also introducing a risk as DNN – the RP – will now accept tokens across portals, but that risk can be mitigated by DNN’s own authorisation.
You can load the audienceURIs section of the configuration in the RP from somewhere else but the web.config (a database table, for example); to do this you would need to add a handler to the FederatedAuthentication.ServiceConfigurationCreated event in the FAM (best way is through the constructor of your custom FAM, InitializeModule is called after the configuration has been loaded) on the RP side and set the audience uri for each portal alias in the DNN database; in a sense this is the RP version of the previous option as it will allow any token to any allowed portal access, even if the token is issued to one portal and the redirect goes to another; it does solve the need to edit the web.config, but it does not solve the need to restart the app domain when changes have been made as the call to the database will only happen once – when the module is initialised.

Both the options above provide some answer to the first approach mentioned at the beginning of this section – allow access to all possible realms, without having to edit web.configs.

The two options below talk about how you could implement a more dynamic check – moving further away from the existing method of checking against a static list of audienceUris -

You can implement your own SamlSecurityTokenHandler (you might want to implement two – one that inherits from Saml2SecurityTokenHandler and another that inherits from Saml11SecurityTokenHandler) in which you would override the ValidateConditions method; in ValidateConditions you would call the base ValidateConditions method, passing in false for the ‘enforceAudienceRestictions’ parameter – this would ensure the configured audienceUris are not checked by the base method; you would then implement your own audienceUri validation, presumably against the DNN database (the conditions parameter passed to you will contain the audienceUri provided by the STS); you could use either code or configuration to setup your RP to use these tokens instead of the built in ones.
A slightly re-factored version of the above is to wrap the validation code required in a custom SamlSecurityTokenRequirement class in which you override the ValidateAudienceRestriction method; both Saml2SecurityTokenHandler and Saml11SecurityTokenHandler classes allow you to provide them with a custom SamlSecurityTokenRequirement class in order to override the built-in logic; this allows you to write the validation logic just once; you will need to replace the default class in the token handlers with yours, which is best done in the same ServiceConfigurationCreated event mentioned above.

DNN’s UrlRewriteModule will redirect any request to the DNN vdirs copying any paths “lower” than the DNN vdir to the query string -

This means that even without dynamically setting the realm and reply address as described above, out-of-the-box you would simply get an infinite redirect situation due to lost cookies -
If you have a portal with the alias www.MyDomain.com/DNN/MyPortal and you set the realm and reply to to this address, you get redirected to the STS and then back to the portal correctly; cookies will be set at the URL above.
However, the UrlRewriteModulre will redirect the request to to www.MyDomain.com/DNN/default.aspx?alias=MyPortal; as the authentication cookies are stored in the original location (one or more virtual directories “lower” than the DNN root directory) they cannot be found and so the user is redirected back to the STS and so on…
The obvious solution is to set the cookie handler path in the configuration to be the DNN’s virtual directory – it would mean that moving between portals would require obtaining a new token and a roundtrip to the STS, but it will solve the circular redirect, which is better (and in any case this is not a likely scenario as far as real users are concerned, as they will normally work on a single portal)

Logout –

Ok – possibly the easiest point encountered – we needed to replace the logout functionality (which would logout the user locally from DNN) to the ws-federation single sign out supported by our STS; to do this we simply replace the code in Login.ascx.vb (in our case it existed in [path to root DNN website]\admin\skins) from the out-of-the-box redirect to a redirect to to the STS with ?ws=wsingout1.0 in the query string.

Going over most items in this list with Jon Simpson – the chief architect in one of the companies I work with – he raised an interesting point – why does every time the Geneva Framework gets upset the user ends up in an endless redirect loop? (ok, he didn’t put it quite in those words, and also generally his view is, naturally, limited to the bits he cares, or – have to worry – about, but the man has a point)

I did not buy into this initially, but I suspect there’s a lot of sense in expecting the framework to recognise there’s an endless loop in place and display some error instead of keeping it going; we will certainly need to do something on our end, as an endless redirect, even if cause by a configuration error on our part, is not acceptable, but this should be part of the framework, or so Jon says :-).

New ‘Oslo’ CTP released, and so there’s a new version of BTSDF

2009-07-21T14:34:00.001+01:00

Somewhat quietly Microsoft released another CTP of ‘Oslo’, which you can find on the Oslo dev centre; it is great to see the team release early and frequent drops of ‘Oslo’, especially given the impact ‘Oslo’ is likely to have on how we build software.

Having the chance to play with the bits so early, and provide feedback, which they seem to be very keen on receiving, is pretty awesome!

I’ve only had a brief play with it and I guess that, to me, the greatest news are that – all the potential benefits aside – my existing code (BTSDF) still worked as is; of course this is only temporary, as the team have made significant changes (read: improvements) to the API, but they have kept backwards compatibility TEMPORARILY as they work to align all their existing code to the new model as the rest of us worry about ours.

So, next I needed to spend some time looking at the new release in detail and align my code with it, so I reap some of the benefits from the improvements made, but that’s a much better position to be in than – “it’s all broken now and I need to figure out how to fix it”m kudos guys!

Paul Arundel was kind enough to give me a gentle nudge to start taking a look at these changes, and it certainly took me longer than I would have wanted to get around to these things thanks to other commitments (a repeating theme here, recently), but – slowly but surely – I went through my code and am happy to say I’ve now published the necessary changes to codeplex.

I was going to write a post on the changes required when moving to the new code, but Paul had done so already, and so there’s little point in me saying pretty much the same things (Paul is only focused on the aspects that are relevant to his project, but as we both focus on pretty much the same area my words would have been pretty much identical).

There only one point I think is worth mentioning from my end – it’s probably only me – but being able to walk to graph in an easier and more convenient way, and being able to easily access nodes by label (or by checking their brand) motivated me to work more on the grammar itself, or – to be more specific – on the projection of the grammar – where previously i would just jump through whatever hoops were needed to parse whatever projection I got.
This is a very good thing I think (although it is the grammar itself that really matters, as this is what users will see); the other half of this, though, is that it is still not quite possible to get the projection just right – I can’t get a projection that would work well with any M model for my domain, for instance, but I suspect the smart guys in Redmond are hard at work sorting this out for us…

Geneva Framework Query String woes

2009-07-09T11:43:00.001+01:00

I bumped into a situation where a query string passed from one RP to another RP was lost as the request went through the STS.

Looking through this carefully I think I figured out why this would happen, and it turns out that the scope of this issue is slightly bigger (and therefore the title of my post is slightly misleading), see the details below - but first – here’s how it goes when everything goes well -

A user tries to access an RP url with some query string parameters; the RP configuration redirects the user to the STS providing realm uri and reply address; as these two values are generally retrieved from the RP configuration, they are unlikely to include the query string parameter.

Good news are, though, that the FAM keeps the original URL requested by the user (excluding the scheme and domain) in a context field passed to the STS with the Sign-in request(the ‘ru’ field).

Once the STS authenticates the user it redirects the request to the reply address provided by the RP; this is NOT the original address the user requested (now in the context) but the reply value provided by the RP through the request, and it will NOT contain the query string parameter.

The FAM, at the RP side, now extracts the URL the user requested from the context and does a second redirect to that URL – the one with the query string.
This time the local cookie is found, the user is authenticated, and the request can be processed by the RP code, where the query string is available to be read.

So – Where is it broken? it appears that if, after all of the above had happened, a request to the RP, with some query string, is redirected again to the STS for authentication for some reason, as the request comes back from the STS to the RP, the SignInResponseMessage is being ignored (as there’s already an authentication cookie) and that second redirect, to the URL in the context does not happen.
In this case, the RP code that will get executed in the page defined in the reply URL in the configuration, which is not likely to have the query string the user requested; in fact – and here’s the bigger scope – it may well be a completely different page to the one the user requested in the browser!

So – the question is – why would I be redirected to the STS when there’s already an authentication cookie for this realm? (after all – if there is cookie I would not be redirected to the STS, right?!) - it turns out there are some ‘edge' cases where this is possible, here’s one -

Step 1

1. Some link somewhere sends the user to the RP using domain name in the URL, with some query string parameters – http://MyDomain.com/SomeApp/Default.aspx?Something=SomethingElse

2.The RP redirects the request to the STS for authentication, providing the realm and reply from the configuration; let’s assume the reply address was set to http://MyDomain.com/SomeApp/SSO.aspx

3. The STS authenticates the user, and redirects back to the reply URL above

4. At the RP, the FAM processes the SignInResponseMessage , stores the authentication cookies for MyDomain.com/SomeApp and then redirects to the context’s http://MyDomain.com/SomeApp/Default.aspx?Something=SomethingElse (note that SSO.aspx never got executed, and that the query string is now available for Default.aspx)

Step 2

5. Now assume that another link somewhere sends the user to the same application, but for whatever reason it uses the server’s IP address instead of the domain – /SomeApp?SomeQueryString">http://<some IP address>/SomeApp/Default.aspx?Something=SomethingElse

6. The RP cannot find any authentication cookies for this URL (as it has the IP address and not the domain name) and so it too, redirects the request to the STS for authentication, however – as its the same configuration, the STS URL, the realm and the reply address are the same as in Step 1, so the STS redirects straight back to http://MyDomain.com/SomeApp/Default.aspx?Something=SomethingElse

7. At the RP, as the URL is now the same as it was in Step 1, the FAM find the previous cookie set and so it ignores the SignInResponseMessge; this means that the redirect that would have happened to the original URL the user requested does not happen, and so it is the page at the reply address, without any query string, that gets processed

There is actually a much simpler way to reproduce the issue, albeit slightly less realistic (post testing) – if you access the RP via it’s IP address, but the reply address in the configuration is set using the domain name;

The cookie will be stored for the domain name, and so any subsequent attempts to access the RP, with query string parameter, will result in them being removed as the redirect at the FAM does not happen.

Implementing Single Sign Out scenario with the Geneva Framework

2009-06-22T20:56:00.001+01:00

One of the items on my to-do list for a while now was to add support for single sign out in our passive scenarios; the idea is that if a user browses to several RPs, and then hits the sign out button on one of them, she would automatically be sign out of all the other RPs visited in this session.

Whilst, as you will see shortly, the framework has great support for this scenario, and it is easily achieved, it is not the out-of-the box behaviour; out of the box – if you’re using the SignInStatus control (with or without the FAM) and the FederatedPassiveTokenService control, when the user hits the sign-out button of the SignInStatus control, she will be signed out of the current RP, as well as the STS itself, but any other RPs the user had visited in this session will keep her logged in.

So – if the user browsed to application A, authenticated at the STS, and then browsed to application B, she is not signed on in both applications as well as on the STS; hitting the sign out button in application B will sign her out of application B as well as of the STS; if she tries to browse to application B now (with no browser caching), she will get redirected to the STS, and would need to re-authenticate there; same would happen if she tried to browse to any application other than application A, which is protected by the STS; within application A, however, the user would still be authenticated and she will be able to keep using this app.

In some cases this may be acceptable, but in our case the users assume that if they hit sign out, they are signed out of the entire “set”, and so we were set on achieving this behaviour.

It turns out that the framework has great support for this scenario, and that only very little code is required to achieve this; in fact – on the RP side – there’s nothing to do.

Both the FAM and the SignInStatus controls handle requests to sign off out of the box, all you have to do is send an HTTP request with “wa=wsignoutcleanup1.0” in the query string and the framework will take care of removing the local cookies; it will even return a nice image to indicate success (you can control which image to show through configuration);

To see this in action – create a standard scenario with two RPs configured to use a single STS; add to your STS an ASPX page, which would look something like this (you will need to update the urls to point at your RPs) -

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="SignOut.aspx.cs" 
                        Inherits="HRG.Profile.Identity.STS.Web.Passive.SignOut" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
                      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<body>
    <form id="form1" runat="server">
        <img src="https://localhost/ststests/testwebsite4/default.aspx?wa=wsignoutcleanup1.0" /><br />
        <img src="https://localhost/ststests/testwebsite3/default.aspx?wa=wsignoutcleanup1.0" />
    </form>
</body>
</html>

In the code behind add the following -

protected void Page_Load(object sender, EventArgs e)
{
    FormsAuthentication.SignOut();
}

Now run through your scenario - login to one application, then browse to another, then browse to this test page; you should see a couple of “green ticks” indicating you have been signed out of both applications.

Now try to browse to either of them (make sure to refresh to pages to avoid browser caching) - you should notice that you’re no longer authenticated in neither (nor the on the STS) and that your’e redirected to the STS’ login page. cool!

So- we’ve proved that there’s really nothing to do on the RP side to achieve single sign out; but what do we need to do on the STS side? well – when the user hits the sign out link button on the SignInStatus control a request goes to the configured url for the STS, so this would be the entry point; what we really need to do is figure out a way to, for example, dynamically generate a page similar to our test page above; to do that we need to be able to a) track the RPs a user had visited and b) control the behaviour of the STS when the user hits sign-out on any RP, to make the required sign out requests to all the other RPs.

Until now I’ve been using the STS control (FederatedPassiveTokenService) in my passive STS, and so – to add behaviour required I would have to extend it, which is not something I felt comfortable doing; the alternative was to get rid of the control altogether and simply write the code required to handle both sign in and sign out from scratch, which is something I wanted to experiment with (in fact – I had to do much of it it in a different area of my solution – bridging single sign out protocols, but that’s for another post), so I though this is a good opportunity to give it a go.

As it turns out, as the framework has all the code to do the heavy lifting, all I needed to do is “control the flow”, and it was all relatively painless - I removed the controls from my page, and started to replace it by placing code in the code behind -

First – I needed to figure out what request I’ve received from the caller; this was as simple as two lines -

WSFederationMessage message = null;
bool messageCreated = WSFederationMessage.TryCreateFromUri(Request.Url, out message);

messageCreated now indicates whether the request to the STS was a valid one, message is expected to be either SignInRequestMessage or SignOutRequestMessage (there are two other possible request types that are not currently supported by the framework, but that’s for another day)

Before I’ll go back to my single sign out scenario, I need to complete the single sign in scenario as I no longer have the control on the page (I could, potentially, leave the control on the page and do nothing if the message was a sign in request – leaving the control to do all the work, and if the message was a single sign out request execute whatever code I needed to achieve that, but I wanted to get rid of the control so I implemented code for both paths)

So – to implement the single sign in my STS needed to call the STS, get a SignInResponse message and write that to the Http response stream, how is this done? well – there may be many favours, but the main code would look something like this (some elements removed for bravity) -

if (message is SignInRequestMessage)
{

   SignInRequestMessage requestMessage = message as SignInRequestMessage;

// Create our STS backend
SecurityTokenService sts = new MySTS(stsConfig);

// Create the WS-Federation serializer to process the request and create the response
WSFederationSerializer federationSerializer = new WSFederationSerializer();
WSTrustSerializationContext serialisationContext = new WSTrustSerializationContext();
// Create RST from the request
RequestSecurityToken request = federationSerializer.CreateRequest(requestMessage, serialisationContext);

// Get RSTR from our STS backend, 
//the thread's principal would not be an IClaimsPrincipal, so create one from the contained identity
IClaimsPrincipal principal = ClaimsPrincipal.CreateFromIdentity(Thread.CurrentPrincipal.Identity);
//issue the RSTR
RequestSecurityTokenResponse response = sts.Issue(principal, request);
// Create Response message from the RSTR
SignInResponseMessage response = new SignInResponseMessage(new Uri(response.ReplyTo), 
                                                           response, 
                                                           federationSerializer, 
                                                           serialisationContext);

response.Write(Page.Response.Output);
Response.Flush();
Response.End();

I’m creating an instance of the STS for each request, but am re-using the sts configuration class (which I keep as an “Application” variable in the STS’ asp.net application).

I’m then using a federation serialiser to create the RST, run this through the STS (providing the principal, “converted” to a ClaimsPrincipal) and then create a SignInResponseMessage out of the RSTR returned by the STS;

Job done – my STS now supports single sign in without the control;

You can already imagine what I needed to do to complete the sign out support- to start with I needed to add an else-if to handle SignOutRequestMessage (as I’ve mentioned – there are other types of requests theoretically possible, but lets not worry about them at the moment) -

else if (message is SignOutRequestMessage)
{

The first thing I would do there, is sign out the user from the STS itslef -

FormsAuthentication.SignOut();

All I needed to do now is add bog standard ASP.net code to generate the required Http Get requests to all my RPs; but to do this I needed to keep track of which RPs a user had visited within the session, so I know which RP’s to sign her out of; to help me achieve that I’ve created the following class to track the user’s visited realms -

public class VisitedRealmsTracker
 {
     private Dictionary<string, string> visitedRealms = new Dictionary<string, string>();

     public void Add(string sessionId, string realm)
     {
         string key = sessionId + "|" + realm;
         lock (visitedRealms)
         {
             if (!visitedRealms.ContainsKey(key))
                 visitedRealms.Add(key, sessionId);
         }
     }

     public IEnumerable<string> GetAllRealmsForSession(string sessionId)
     {
         //find all visited realms for this session and return the second part of they key (after the '|') which would be the realm
         return from visitedRealm 
                in visitedRealms 
                where visitedRealm.Value == sessionId 
                select visitedRealm.Key.Split('|')[1];
     }

     public void ClearUserEntries(string sessionId)
     {
         lock (visitedRealms)
         {
             List<string> keys = 
                 visitedRealms.Where(realm => realm.Value == sessionId)
                              .Select(realm=>realm.Key).ToList();
             foreach (string key in keys)
                 visitedRealms.Remove(key);
         }
     }
 }

I’ve added a member of this type to my STS Configuration class, which – you would remember – I now keep as an application variable, and so I could add a call to Add in the STS code handling the sign in request I showed earlier, which would ensure I’m tracking all the visited realms; the sign out logic could now iterate over the results of the GetAllRealmsForSession and handle the logout requests, simple code to achieve this could be something like -

foreach(string realm in stsConfig.Tracker.GetAllRealmsForSession(Session.SessionID))
{
    //get realm configuration
    ReliantPartyConfigurationElement rpConfig = Configuration.ReliantParties[realm];
    //create an image pointing the at realm's signout url appending the signout and cleanup action
    Image img = new Image();
    img.ImageUrl = rpConfig.SignOutUrl.Trim()+"?wa=wsignoutcleanup1.0";
    Repeater1.Controls.Add(img);
    //add a line break after each realm
    LiteralControl br = new LiteralControl("<br />");
    Repeater1.Controls.Add(br);
}

This sample code simply creates the same images I’ve previously had hard coded in the test page dynamically.

With all the code in place – sign in requests are being processed by code instead of the control, with the code now customised to keep track of visited realms, sign-out requests use this tracked information to dynamically build a page that issues the required Http get request to all the visited RPs to sign the use out of all of them; single sign out achieved. easily.

Geneva Framework and Url case sensitivity- solved?

2009-06-12T11:26:00.000+01:00

I’ve blogged before (somewhat briefly, for a change) about my surprise when I learnt that URLs are [largely theoretically, in my view] case sensitive and the problem that this causes for a Geneva Framework based passive STS implementation.

In that post I mentioned a solution suggested by Peter Korn at the time – setting the path of the cookie to the domain root (‘/’) instead of the application path (including virtual directories), as, unlike the rest of the path, the domain name in a URL is not case sensitive, this works well, and I though it was “case closed”; until recently, when I’ve realised this solution has a very significant drawback - as the cookie, containing the authorisation token from the STS, is stored at the root of the domain, it will be served to every application under that domain, which is taking single-sign-on slightly too far :-)

Following this approach it is not possible allow access to one application and deny it from another (on the same domain) other than through claims processing in the applications themselves, which is a less secure approach from an architecture perspective); clearly not a good solution then…

So – I needed to go back to storing the cookie in the correct path, which would ensure that the STS is re-visited when trying to access a second application (even in the same domain), which – in turn – would mean that the user’s permissions are re-evaluated, before a second, application-specific, token is provided; with that - came back the problem of the URLs being case sensitive.

Thankfully, we’re now on the TAP program for the Geneva Framework, and we’re getting great support by the guys at Redmond (can’t thank them enough!), and after bringing up this issue in a discussion, Shiung Yong suggested another approach to solving this - overriding the GetReturnUrlFromResponse method in the WSFAM.

(Side track: The more I work with the Geneva Framework the more impressed I am with the extensibility options it provides, sure – it’s hard to figure those out on your own if you don’t know about them, and yeah – the resulting solution is often somewhat fragmented, with bits of code in several places, but that’s not much different from many other solutions in this space I suspect – you can see this with many WCF implementations – on the upside, however, if you’re willing to put the sweat, you can do pretty much everything (but yes – the continuum moves from adding a couple lines of code to re-writing the framework :-) )

To understand why and how Shiung’s solution works, consider the following scenario, describing the problem (and here’s where my description is bound to get somewhat confusing) -

Out of the box, the flow of circular redirects, when the URL in the browser is entered in the “wrong” casing, is as follows -

The user types in the RP’s URL, let’s say - all in uppercase, into the browser
As the http request to the RP does not contain an authentication token at this point, the FAM at the RP redirects it to the STS, providing the RP’s ‘realm’ to the STS (the ‘realm’ is configured at the RP and is intended to provide a unique URI to the STS, which it can use to identify the RP, and, for example, be used to load the relevant configuration such as which certificates to use when creating the token); the original URI the user had typed in is also provided through the query string (the ru property in wctx); optionally, and crucially, the RP may also provide the wreply query string parameter, based on its configuration; it is expected that the STS will forward the request, after authentication the user, to this address (but this is not mandated), this will become a key point shortly.
Still at the STS the user authenticates (generally using a login screen), and the STS redirects the request, with a ‘sign in response’ message containing an authentication token back to the RP; as mentioned before it is expected that this would be the address provided by the wreply (and this would be the default behaviour provided by the framework, but this can be easily overridden in the STS’ implementation); for this example, lets assume that the configured value, echoed in the wreply property is set to be in lowercase (remember – the user typed in the URL in the browser in uppercase).
The redirect request contains the set-cookie instructions with the token from the STS and so the browser would set the required cookies in the address the STS redirected to - the lowercase address.
In the step that would follow, the FAM does its sign-in ‘magic’, which concludes by redirecting the request to the URL set in the STS’ response message through the ru field in the query string - this is the URL the user typed into the browser initially, kept by the RP and then the STS - which is all uppercase
At this point, FAM is called again for the new request, attempting to extract the authentication cookie, but as the cookie was stored on the URL the STS redirected to – which was lowercase - and the browser is now using the URL the user typed initially – which is all uppercase - the cookie is not served by the browser, and thus not found in the server code, and the user is being redirected back to the STS as if this was the first call;
As the request arrives to the STS with the uppercase url again, the above would happen again and again in an endless cycle.

Confused? hopefully not too much…but to summarise - out of the box, if the two (the URL configured as the reply to address in the RP, or any other URL the STS uses to redirect back to the RP) and the URL typed into the browser by the user) are not [case-sensitive] identical, the cookie will be set, but subsequently not found when attempted to be read and thus authentication at the RP would continuously fail.

In comes Shiung’s solution -

As long as there’s a convention in the implementation as to the correct form of the URLs (or if drowning in more configuration is acceptable) the FAM could be extended to over come this -

Step 5 above mentions the FAM has some ‘magic’ authentication work with a redirect in the end; the built in implementation uses the ru field to obtain the address to redirect to, but there’s a good extension point there in the form of the GetUrlFromResponse method of the FAM which is called to obtain the url; by overriding this function you can provide whatever logic you wish to control the URL the FAM would redirect the request to after authenticating the request.

Lets say we can agree (as we have) that all reply to addresses will always be configured in lowercase (whilst we can’t control user behaviour, we can control our own configuration), with that agreed we can override the GetUrlFromResponse to always convert the ru value to lowercase before returning it to the bulit in functionality – here’s my version of the method, as suggested by Shiung -

   public class CaseInsensitiveFAM : Microsoft.IdentityModel.Web.WSFederationAuthenticationModule
    {
        protected override string GetReturnUrlFromResponse(System.Web.HttpRequest request)
        {
            string returnUrl = base.GetReturnUrlFromResponse(request);
            return returnUrl.ToLower();
        }
    }

(it is important, of course, to remember to configure the RP to use your custom FAM and not the build-in one -

        <!--<add name="WSFederationAuthenticationModule"
                 type="Microsoft.IdentityModel.Web.WSFederationAuthenticationModule, 
                       Microsoft.IdentityModel, Version=0.6.1.0, Culture=neutral, 
                       PublicKeyToken=31bf3856ad364e35"/>-->
        <add name="CaseInsensitiveFAM" type="CaseInsensitiveFAM, Utilities"/>

)

What had just happened?

By convention, we ensured the RP provided a lowercase reply to address to the STS.

The STS uses this (lowercase) address to forward the request containing the authentication token, and this is where the cookies will be set.

The FAM uses GetUrlFromResponse to retrieve the URL to redirect to, my customised version ensures this would always be lowercase, aligned with the RP configuration

The browser is redirected, again to the lowercase address, this time to receive the cookies set in step 2 which means the request is now authenticated and the user is let in; no more circular redirects!

Of course I’ve implemented a hardcoded rule (always lowercase), but you could use configuration, investigate the http request message or any other logic you’d like…

Some issues remain with that approach (that I can think of) -

If, at this point, the user goes and types the URL in a different casing, as the cookie already exists and the FAM code will not execute again, the user will get redirected to the STS for authentication, but that’s fair enough – I don’t know of any user that would do that in real life..and the result (requiring re-authenticaiton) is quite acceptable

The other thing is that this solution would break should an application be case sensitive (for query string parameters, for example), but we don’t have that problem, and it could be handled by more sophisticated code in the custom FAM, so that’s ok as well.

I suspect this is not the clearest post I’ve ever published (but, unfortunately, probably not the worst), so I can only hope someone will manage to make sense of it and find it useful; I’m pretty sure I’ll need it for future reference; there’s no chance I’m remembering all of this!

Oslo based solution for deploying BizTalk applications – published

2009-05-26T09:35:00.001+01:00

Unfortunately life is a bit hectic for us at the moment, and it’s been a while since I’ve posted anything, or was able to do anything other than family or work.

As part of this general “neglect” I was unable to spend the time required to complete, and publish, my Oslo based solution for deployment BizTalk applications (given the somewhat ugly, but suitably short, name – BTSDF), but as many waited (ok – more than 5), and some kept asking (2), I thought I’d do my best to get something out of the door, so I have.

in Codeplex you can now find an initial version published, which includes – the language definition, the core runtime I have as well as two “executors” in various stages of [non] completeness – my MsBuild generator is already quite useful for simple apps, it generates a set of MSbuild files you can use to deploy the application on any machine as well as the required SDC tasks dlls and targets file; the API executor deploys the app on the current machine, and is quite basic, but a good sample (I think) and a reasonable starting point.

I’m happy to entertain requests for changes/additions, and even more happy to add any one who’s willing and able to put the time as a contributor; Oslo knowledge is optional! :-)

You can download the source, build and run it locally – but you will have to remember to copy the two executors dlls from their folders to the main bin\debug folder as there is no compiled reference to them.

I’ve also uploaded an “Alpha release” which includes the compiled assemblies and the supporting files required.

There’s still more work to do, but it’s getting shape now, and I’m using the MSbuild executor today; it certainly needs a bit more documentation, which I hope to get around to…

I hope this works for you, let me know what you think (the good and the bad), and – if you have some time and will – drop me a line and I’ll add you to the team!

Yossi

RoleClaimsMapper for the Geneva Framework

2009-04-01T08:35:00.001+01:00

Briefly back on my STS work -

Our STS implementation can already replace the authentication implementation of most of our applications; naturally we can’t do that just yet, given that the Geneva-framework has not been released yet, but all of my tests are quite positive so we’re just waiting for the opportunity to start using it.

However, so far, we were not in a position to replace the authorisation mechanism, not easily anyway, and that’s something that was on my list for some time now.

The STS provides a list of claims, which the applications can relatively easily access via code, as many samples show, and this proves very useful; application can investigate various claims about a user and drive their functionality from that.

It does mean, though, that the applications need to change to support this new claims based mode for authorisation, which is not something we can just assume we would be able to do; as a start, we just want to achieve an in-place replacement for our current authorisation logic.

Most of our web apps currently use ASP.net membership and roles and so they extensively use ‘IsInRole’ checks to figure out user authorisation and drive the application behaviour, to start with, we had to hook to that mechanism.

Luckily the Geveva framework has a relatively good support for exactly this need - , out of the box, it would convert any claims of the Microsoft role namespace (‘http://schemas.microsoft.com/ws/2008/06/identity/claims/role’) to roles; so – if a token included a claim of this type with a value of ‘Manager’, a call to HttpContext.Current.User.IsInRole(“Manager”) would return true.

And so I made sure my STS adds any roles with the correct claim type, very easy.

However – this is very Microsoft centric. what about all those claims that come from systems that don’t follow Microsoft’s approach? (how dare they!) ? and what about us wanting to have our own claims, using our own types, some matching roles (while others may not…) -

Well – we needed a way to map any claims to ms-role claims before the Geneva framework does its bit.

As is often the case - Dominick Baier was most helpful in posting on exactly that, and so, following his example, I created my RoleClaimsMapper -

    public class RoleClaimMapper : ClaimsAuthenticationManager
    {
        public override IClaimsPrincipal Authenticate(string endpointUri, IClaimsPrincipal incomingPrincipal)
        {
            //load configuration section for component
            RoleClaimsMapperConfigurationSection config = 
                (RoleClaimsMapperConfigurationSection)ConfigurationManager.GetSection("RoleClaimsMapper");
            //create a collection of claim types and populate from configuratoin
            List<string> claimsToMap = new List<string>(config.RoleClaims.Count);
            foreach (RoleClaimConfigurationElement claimElement in config.RoleClaims)
                claimsToMap.Add(claimElement.ClaimType);
                
            //loop on all identities, we really only expect one, but can easily support multiple.
            foreach (IClaimsIdentity identity in incomingPrincipal.Identities)
            {
                //extract the claims that we need to map (matching the configured list of claims)
                IEnumerable<Claim> roleClaims = 
                    identity.Claims.Where<Claim>(c => claimsToMap.Contains(c.ClaimType));
                //now create a role claim (using the MS role claim type) for each claim found; 
                //need to keep this outside claim loop so we don't modify the collection while iterating
                List<Claim> claimsToAdd = new List<Claim>(roleClaims.Count()); 
                foreach (Claim claim in roleClaims)
                    claimsToAdd.Add(new Claim(Microsoft.IdentityModel.Claims.ClaimTypes.Role, claim.Value,claim.ValueType,"local",claim.Issuer));
                //add new claims to current identity
                identity.Claims.AddRange(claimsToAdd);
            }

            return incomingPrincipal;
        }
    }

I then configured my authentication manager with the framework -

<microsoft.identityModel>
    <claimsAuthenticationManager type="RoleClaimMapper,Identity.Utilities" />

and added my bit of custom configuration

  <RoleClaimsMapper>
    <RoleClaims>
      <add Type="http://someDomain.com/identity/claims/SomeRole"/>
      <add Type="http://someDomain.com/identity/claims/AnotherRole"/>
    </RoleClaims>
  </RoleClaimsMapper>

As you can see this code would take a list of claim types from configuration, and map all claims of these types to roles, adding them to the identity’s claims collection using the required claim type (leaving the original claim intact), and voila – when the app executes it can check the roles, corresponding to the values supplied in my custom claims using -

HttpContext.Current.User.IsInRole(“[custom claim value"]”);

Very nice indeed!

UPDATE: Dominic’s comment on this post (on http://geekswithblogs.net/Connected/archive/2009/04/01/roleclaimsmapper-for-the-geneva-framework.aspx#453535) had indirectly suggested an even cleaner solution; instead of duplicating the claims, I can add the claim types I have as roles to each identity’s RoleClaimTypes collection; this achieves the same result in a much cleaner way, here is the updated function -

        public override Microsoft.IdentityModel.Claims.IClaimsPrincipal Authenticate(string endpointUri, Microsoft.IdentityModel.Claims.IClaimsPrincipal incomingPrincipal)
        {
            //load configuration section for component
            RoleClaimsMapperConfigurationSection config = 
                (RoleClaimsMapperConfigurationSection)ConfigurationManager.GetSection("RoleClaimsMapper");
                
            //loop on all identities, we really only expect one, but can easily support multiple.
            foreach (IClaimsIdentity identity in incomingPrincipal.Identities)
                //for each identity, add all the claim types that are role claim types.
                foreach (RoleClaimConfigurationElement claimElement in config.RoleClaims)
                    identity.RoleClaimTypes.Add(claimElement.ClaimType);

            return incomingPrincipal;
        }

Oslo based solution for deploying BizTalk applications – the runtime

2009-03-29T09:51:00.001+01:00

This is a third post in a series describing my Oslo based solution for deploying BizTalk applications; I’ve used this exercise to play around with ‘M’, but it was important for me to work on a real solution, with real benefits – something I could actually use…in Part I I discussed the concept and presented both the “source code” of my app and the output I was working toward; Part II was all about the MGrammar part of the solution.

In this, third, part I will discuss the last missing piece –the runtime.

Before I start, though, I would say that I did find getting into ‘M’ somewhat confusing at first; and while it’s more than just possible I’m still missing some things , I hope this series could help one or two people in their journey with Oslo – which is, without a doubt, an exciting one!

There are two things, I believe, that contributed to my confusion - the first is the fact the M is really many things, quite different things, actually; from what I hear Microsoft have identified the challenge some of us (me) are having getting a grasp on ‘M’ and are hard at work to bring things [closer] together; hopefully it won’t be long before we know how the converged language looks like, in the mean time one simply has to remember that -

There’s MSchema - which you could use to define models, a bit like xml-schema, or declaring your classes in code or even tables in SQL; I haven’t really touched on MSchema in this series butI might come back to that later.

Then there’s MGraph, which is a way to define instances of things, possibly ones that have been modelled using MSchema, but, as is evident from my little project, not necessarily - MGraph can be very useful even if you don’t have a model- as long as you have your grammar – in comes MGrammar, the third spect of ‘M’, which can be used to define a syntax for your very own [domain-sepcific-]languge for describing things;
A ‘runtime’ could then be used to processes instances described as MGrammar as a result of inputs in your language.

And that is the second thing that really confused me – what is that ‘runtime’? in all the ‘M’ presentations I’ve seen, the ‘runtime’ was merely mentioned and has never received enough “floor space” and yet – an MGrammar without a runtime, in the majority of cases, is, quite useless; you have to have a runtime that would act on your source code; in fact – the runtime would act on the MGraph resulting from your language, which is what makes it all so brilliant, because in a sense, this is where everything comes together – you runtime can work on instances described in your language, on MGraph instances stored in the repository created using MSchema and possibly even ones defined using Quadrant.

The point is that there must be a runtime that understand the model behind your language , can parse its graph and then do whatever you need it to do; and it is your job to build that runtime.

So what have I done for my runtime? here’s a quick overview (reminder: the full source code will find its way shortly onto codeplex) -

My runtime is a console application, one that takes a source code file path as an argument and outputs MSBuild files (and dependencies) that can be used to deploy the application described in the source code onto BizTalk Server.

The first part of my runtime - which I will not bore you with - is about validating the command line arguments; standard stuff.

The second part is about creating the parser for my language, where, thankfully, the Oslo SDK does all of the heavy lifting – it includes a class called DynamicParser which, once created, you can use to parse your source code.

To create the DynamicParser you must first compile your language, and that’s easy enough to do – you start by creating a compiler

MGrammarCompiler compiler = new MGrammarCompiler();

and continue by supplying your grammar

compiler.SourceItems = new SourceItem[] {
        new SourceItem {
            Name="BTSDeploy",
            ContentType = ContentType.Mg,
            TextReader = new StreamReader(GetLanguageDefinition())
        }
    };

(GetLanguageDefinition() is a simple helper method I wrote to get the grammar file embedded as a resource in the exe)

Now you’re ready to compile your language, but to make things manageable you want to provide it with an error reported; the compiler would report any errors to the stream you would provide, I’ve naturally used the console

TextWriterReporter errorReporter = new TextWriterReporter(Console.Out);
if (compiler.Compile(errorReporter) != 0 || errorReporter.HasErrors)
{
    Log("Failed to compile language definition\nSee above for details");
    return null;
}

If the compilation succeeded you are ready to create your parser -

DynamicParser parser = new DynamicParser();
compiler.LoadDynamicParser(parser);

That’s part one of three done.

The next step is to use the dynamic parser to parse your source code, the output of which would be a graph representation of the source; luckily the SDK does virtually all the lifting here as well, and it comes down to one line -

object rootNode = parser.Parse<object>(sourceCodeFileName, null, errorReporter);

Note that the output type is object – which, as you will find out if you try this out, is quite painful– currently all the types used in the Graph are internal, which makes debugging quite difficult (you can’t quite look at any variables you hold in any meaningful way, you have to keep calling methods, as you’ll see next; hopefully this will change one of the next updates to the SDK.

In any case rootNode is now pointing at the root of a graph – a tree like structure you could ‘walk’ to extract the pieces of information you care about in the source code; here you’re expected to use methods like GetLabel, GetSequenceElements and GetSuccessors to reach nodes and their values in the graph and, of course, to do that you need to know exactly how your graph looks like; my first instinct was to look at the PreviewMode pane in intellipad (usually the right most pane when working with MGrammar) as it shows you a representation of the MGraph created for the source code and language used; this worked quite well, but, as I found out, wasn’t the most trivial thing – the two didn’t align completely and I ended up having to resort to trail-and-error to get the parsing logic right.

The reason is that M has a few shortcuts one could take, but the graph you would be working on is the very basic, more verbose format; some information on this is mentioned here.

Then, on a recent visit to Redmond, Dana Kaufman passed on a great tip – if you ‘compile’ your grammar using mg.exe to create the mgx file (basically a ZIP file containing XAML representation of language) and then use mgx.exe on your source file adding a reference to the mgx file you just created, you end up with an ‘M’ file which is exactly the graph your runtime would be working on.end up with; so useful!

So – here are a few example of how I worked the graph – to start with I knew my root node should be a node with the label ‘Application’, so I checked it this way -

string label = graph.GetLabel(rootNode).ToString();
if (label != "Application")

I then knew that the application name would be a child element of the root node, so I extracted it like this

// extract the application's data - this should contain two nodes - the application name and the list of items in the application
List<object> appData = graph.GetSequenceElements(rootNode).ToList<object>();
//first line should be the application name, make sure it is not a node and extract the label
if (!graph.IsNode(appData[0]))
    Contents.AppName = appData[0].ToString();

the second node in the appData collection is where the graph ‘continues’, so to get the list of things that compose my application I needed to walk down that path -

//the second element should be the list of lines
foreach (object section in graph.GetSuccessors(appData[i]))
{
  //each successor would be a category (reference, importing binding, resource, etc), with a list of items
  processSection(graph,section);
}

with processSection start with

string sectionName = graph.GetSequenceLabel(graph.GetSuccessors(section).First()).ToString();
List<object> items = graph.GetSuccessors(graph.GetSuccessors(section).First()).ToList<object>();

Log("Found section '{0}'", sectionName);
switch (sectionName)

I hope that from these few examples you can see what it takes to work the graph – the graphBuilder (which is a somewhat confusing name, as I’m using it to walk the graph, not build it) has all the methods you need to access the various nodes ( but there’s no xpath-like- support), but as all the types are (currently) internal to the MS assembly you’re always working with objects, which is less then ideal.

Again – my full source code is on its way to codeplex, I just want to make sure it’s commented well enough to be well understood, and am struggling with time, but the bottom line is that once you figure out how the graph builder works, learnt how to see your graph visually (using mg.exe and mgx.exe) and got used to the fact that you’re dealing with objects for now, parsing the source code is very easy.

Obviously it is completely down to you what you then do with all the information you’ve extracted from the source code; in my case my runtime is using a plug-in model so the first part is all about using the Oslo SDK to get an instance of a BizTalkDeployment class populated based on the contents of the input file, this class looks like -

    public class BizTalkDeployment
    {
        public string AppName { get; set; }
        public List<object> References { get; set; }
        public List<object> Build { get; set; }
        public List<BizTalkAssembly> BizTalkAssemblies { get; set; }
        public List<object> ImportBindings { get; set; }
        public List<Binding> AddBindings { get; set; }
        public List<Assembly> Resources { get; set; }
    }

I then use late binding and configuration to load a plug in that would take an instance of this class and do the work, be it generation of msbuild scripts, deploying to the local machine using BTSTask or anything else.

On Atomic Scope and Message Publishing

2009-03-23T09:15:00.001Z

A few weeks back I worked on a process that looked something like this -

It was triggered by the scheduled task adapter and then used a SQL send port to call SP to return list of ‘things’.
It needed to split the things in the list to individual records, and to start a new, different, process, through pub/sub (to avoid the binary dependency with the called process), for each ‘thing’.

Fairly simple.

A lot of have been said on the different ways to split messages, I won’t repeat this discussion here; I would just say that initially I used a different approach – I used the SQL adapter in the initial, triggering, receive port and then used a receive pipeline, with an XmlDisassembler component, to split the incoming message so that each record was published individually thus avoiding the need to have a ‘master process’; that back fired though, in my case – I quickly realised I’ll be choking the server with the amount of messages published and needed a way to throttle the execution; I’ve played a bit with host throttling but then came to the conclusion the best approach for me would be to throttle in a process, which is what I’ve done.

And so - to make things interesting, and because I already had it all ready - I decided to use a call to a pipeline from my process to split the message.

The first thing I realised, trying to take that approach, was that I had to change type of the response message received from the SQL port to be XmlDocument (which is an approach I generally dislike – I’m a sucker for strongly-typed-everything) – but my schema was configured as an envelope so that when I call the pipeline from my process it knows how to split it correctly, but, when used in the SQL port BizTalk split the message too early for me – I needed to whole message in the process first, which was no good to me; if , however, I removed the envelope definition from the schema when I would call the pipeline directly from my process it won’t know how to split the message, which is no good either; nor could i have two schemas (BizTalk, as we all know, dones’t like that bit at all, not without even more configuration); XmlDocument it is.

It then came back to me (in the form of a compile time error :-)) that the pipeline variable has to exist in an atomic scope, and so I added one to contain my pipeline variable; I then added the necessary loop with the condition set to the GetNext() method of the pipeline and in each iteration constructed a message using the GetCurrent() method; all standard stuff.

I would then set some context properties to route my message correctly and allow me to correlate the responses (I used a scatter-gather pattern in my master process) and published it to the message box

What I noticed when testing my shiny new process was that all those sub-processes that were meant to start as a result the published messages in my loop were delayed by quite a few minutes (6-8), which seemed completely unreasonable, so I embarked on a troubleshooting exercise which resulting in that big “I should have thought of that!” moment.

While the send shape in my loop successfully completed its act of publishing the message in each iteration, moving my loop to the next message and so on, being in an atomic scope BizTalk would not commit the newly published messages to the message box database, allowing subscriptions to kick in, before the atomic scope would finish; that is to allow it to rollback should something in the atomic scope would fail.
What it meant for me though, was that all the messages were still effectively published at once, which brought me back to square one (or, minus one, actually, considering that the great delay caused my this approach means I’m even worse off from my first debatch-in-pipeline approach).

And so I went back to the old and familiar approach of splitting the messages using xpath in the process, which allowed me to carefully control the publishing rate of messages for my process and throttle them as needed.

ASMX, WCF and enums woes

2009-03-18T10:17:00.001Z

We’ve been slowly migrating our services from asmx to WCF, but as we’re still using BizTalk 2006 with no support for WCF we’ve been exposing endpoints configured for basicHttpBinding and consume them using the SOAP adapter.

Generally speaking things have been going well, although we completely gave up on the idea of moving the services to WCF and NOT have to change the client, until yesterday we’ve stumbled into a serialisation issue –

The SOAP adapter, as part of its work deserialises the request message arriving through the send port into t he web service proxy class it generated, before calling the web service (which would result in the class now being serialised back into xml, which is another story); that deserialisation failed.

The error message was clear enough and indicated it failed to deserialise an enum parameter the service was expecting, and that ran a bell – I posted on exactly that back in September, but after carefully checking and re-checking everything we could swear that our message (which was now suspended) matches perfectly the schema generated by the add web reference wizard; what’s going on then??

After chasing our tail for a short while we brought up reflector to the rescue and found out the cause of our woe is a combination of a difference in behaviour between WCF and ASMX and the use of BizTalk – here are the details –

Consider the following asmx web method –

[WebMethod]
 public string GetDataUsingDataContract(CompositeType.someEnum myEnum)
 {
     return "Hello World";
 }

With CompositeType being

public class CompositeType
 {
   public enum someEnum
   {
      Value1,
      Value2
   }
 }

(..and pretend CompositeType has many more things, but these are irrelevant to this topic)

The definition for myEnum in the WSDL looks like

<s:element minOccurs="1" maxOccurs="1" name="myEnum" type="tns:someEnum" />

Where the type tns:someEnum looks like

<s:simpleType name="someEnum">
    <s:restriction base="s:string">
    <s:enumeration value="Value1" />
    <s:enumeration value="Value2" />
    </s:restriction>
 </s:simpleType>

As a result the definition of the enum in a proxy generated via the add web reference VS 2005 option (which is what BizTalk would use) looks like –

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "2.0.50727.3053")]
 [System.SerializableAttribute()]
 [System.Xml.Serialization.XmlTypeAttribute(Namespace="http://tempuri.org/")]
 public enum someEnum

 {
     Value1,
     Value2,
 }

All makes sense.

Now, let’s look at what WCF does in the same case; consider the following service –

[ServiceContract]
 public interface IService1
 {

   [OperationContract]
    string GetDataUsingDataContract(CompositeType.someEnum myEnum);
 }

 [DataContract]
 public class CompositeType
 {

    public enum someEnum
    {
        Value1,
        Value2
    }
 }

The WSDL generated looks like

<xs:simpleType name="CompositeType.someEnum">
 <xs:restriction base="xs:string">
   <xs:enumeration value="Value1" />
   <xs:enumeration value="Value2" />
 </xs:restriction>
 </xs:simpleType>
 <xs:element name="CompositeType.someEnum" nillable="true" type="tns:CompositeType.someEnum" />

The key difference is that the name of the class containing the enum has made it into the type name for the enum, which never happened in the ASMX version.

As a result the proxy is generated as such -

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Runtime.Serialization", "3.0.0.0")]
 [System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                  Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value2 = 1,
 }

Again – note the name given to the element now contains the class name and, crucially, a dot (‘.’).

On it’s own – nothing to malicious – although it’s another nail in the coffin for the idea that you can substitute web service with WCF service, configured them to use basicHttpBinding and all should be the same (ok – am I the only one still wishing this was possible?)

Enters BizTalk.

When you use the add web reference wizard to add a reference to the WCF service, BizTalk generates all the schemas and proxy for you, which is what you would use to create requests going to the service (and process responses).

Because the WSDL of the WCF service contains the longer name of the enum (with the class name, the dot and the enum name) the .net proxy generated is identical to the one created for the WCF service above; the schema, however, is generated incorrectly!

BizTalk “kindly” decides that having dots in the element name is not a good idea and removes it so the schema generated looks like this –

<xs:schema xmlns:tns="http://schemas.datacontract.org/2004/07/WcfService1" elementFormDefault="qualified" 
          targetNamespace="http://schemas.datacontract.org/2004/07/WcfService1" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element name="CompositeTypesomeEnum" type="tns:CompositeTypesomeEnum" />
 <xs:simpleType name="CompositeTypesomeEnum">
   <xs:restriction base="xs:string">
      <xs:enumeration value="Value1" />
      <xs:enumeration value="Value2" />
   </xs:restriction>
 </xs:simpleType>
</xs:schema>

“CompositeTypesomeEnum”??????

Well, we’ve seen this, and created a message with exactly that element, which – of course – the SOAP adapter failed to deserialise into

[System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                                                    Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value2 = 1,
 }

The solution was fairly simple – we’ve simple change our xsl to put the element name as the .net proxy requires it, and not as the schema describes it, and it all worked well.

Ok, got your message, now let us have xsl re-use!

2009-03-16T08:46:00.001Z

From the Microsoft Knowledgebase -

The <xsl:import> element is used to import an external XSLT file. The <xsl:include> element is used to include an external XSLT file. You cannot use these elements in custom XSLT files that are defined in the Custom XSL Path Grid Properties in a BizTalk project. You cannot do this because a Uniform Resource Identifier (URI) that is defined in an external XSLT file may be from a nonsecure source such as the Internet.

Am I (well, and Ben Gimblett here) the only one who thinks this is a lame excuse? since when MS tries to protect developers from stupidity? and in any case, if they really wanted to do that – wouldn’t they have to prevent us from writing ANY code?