Yossi Dahan [BizTalk]

Implementing Single Sign Out scenario with the Geneva Framework

2009-06-22T20:56:00.001+01:00

One of the items on my to-do list for a while now was to add support for single sign out in our passive scenarios; the idea is that if a user browses to several RPs, and then hits the sign out button on one of them, she would automatically be sign out of all the other RPs visited in this session.

Whilst, as you will see shortly, the framework has great support for this scenario, and it is easily achieved, it is not the out-of-the box behaviour; out of the box – if you’re using the SignInStatus control (with or without the FAM) and the FederatedPassiveTokenService control, when the user hits the sign-out button of the SignInStatus control, she will be signed out of the current RP, as well as the STS itself, but any other RPs the user had visited in this session will keep her logged in.

So – if the user browsed to application A, authenticated at the STS, and then browsed to application B, she is not signed on in both applications as well as on the STS; hitting the sign out button in application B will sign her out of application B as well as of the STS; if she tries to browse to application B now (with no browser caching), she will get redirected to the STS, and would need to re-authenticate there; same would happen if she tried to browse to any application other than application A, which is protected by the STS; within application A, however, the user would still be authenticated and she will be able to keep using this app.

In some cases this may be acceptable, but in our case the users assume that if they hit sign out, they are signed out of the entire “set”, and so we were set on achieving this behaviour.

It turns out that the framework has great support for this scenario, and that only very little code is required to achieve this; in fact – on the RP side – there’s nothing to do.

Both the FAM and the SignInStatus controls handle requests to sign off out of the box, all you have to do is send an HTTP request with “wa=wsignoutcleanup1.0” in the query string and the framework will take care of removing the local cookies; it will even return a nice image to indicate success (you can control which image to show through configuration);

To see this in action – create a standard scenario with two RPs configured to use a single STS; add to your STS an ASPX page, which would look something like this (you will need to update the urls to point at your RPs) -

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="SignOut.aspx.cs" 
                        Inherits="HRG.Profile.Identity.STS.Web.Passive.SignOut" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
                      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<body>
    <form id="form1" runat="server">
        <img src="https://localhost/ststests/testwebsite4/default.aspx?wa=wsignoutcleanup1.0" /><br />
        <img src="https://localhost/ststests/testwebsite3/default.aspx?wa=wsignoutcleanup1.0" />
    </form>
</body>
</html>

In the code behind add the following -

protected void Page_Load(object sender, EventArgs e)
{
    FormsAuthentication.SignOut();
}

Now run through your scenario - login to one application, then browse to another, then browse to this test page; you should see a couple of “green ticks” indicating you have been signed out of both applications.

Now try to browse to either of them (make sure to refresh to pages to avoid browser caching) - you should notice that you’re no longer authenticated in neither (nor the on the STS) and that your’e redirected to the STS’ login page. cool!

So- we’ve proved that there’s really nothing to do on the RP side to achieve single sign out; but what do we need to do on the STS side? well – when the user hits the sign out link button on the SignInStatus control a request goes to the configured url for the STS, so this would be the entry point; what we really need to do is figure out a way to, for example, dynamically generate a page similar to our test page above; to do that we need to be able to a) track the RPs a user had visited and b) control the behaviour of the STS when the user hits sign-out on any RP, to make the required sign out requests to all the other RPs.

Until now I’ve been using the STS control (FederatedPassiveTokenService) in my passive STS, and so – to add behaviour required I would have to extend it, which is not something I felt comfortable doing; the alternative was to get rid of the control altogether and simply write the code required to handle both sign in and sign out from scratch, which is something I wanted to experiment with (in fact – I had to do much of it it in a different area of my solution – bridging single sign out protocols, but that’s for another post), so I though this is a good opportunity to give it a go.

As it turns out, as the framework has all the code to do the heavy lifting, all I needed to do is “control the flow”, and it was all relatively painless - I removed the controls from my page, and started to replace it by placing code in the code behind -

First – I needed to figure out what request I’ve received from the caller; this was as simple as two lines -

WSFederationMessage message = null;
bool messageCreated = WSFederationMessage.TryCreateFromUri(Request.Url, out message);

messageCreated now indicates whether the request to the STS was a valid one, message is expected to be either SignInRequestMessage or SignOutRequestMessage (there are two other possible request types that are not currently supported by the framework, but that’s for another day)

Before I’ll go back to my single sign out scenario, I need to complete the single sign in scenario as I no longer have the control on the page (I could, potentially, leave the control on the page and do nothing if the message was a sign in request – leaving the control to do all the work, and if the message was a single sign out request execute whatever code I needed to achieve that, but I wanted to get rid of the control so I implemented code for both paths)

So – to implement the single sign in my STS needed to call the STS, get a SignInResponse message and write that to the Http response stream, how is this done? well – there may be many favours, but the main code would look something like this (some elements removed for bravity) -

if (message is SignInRequestMessage)
{

   SignInRequestMessage requestMessage = message as SignInRequestMessage;

// Create our STS backend
SecurityTokenService sts = new MySTS(stsConfig);

// Create the WS-Federation serializer to process the request and create the response
WSFederationSerializer federationSerializer = new WSFederationSerializer();
WSTrustSerializationContext serialisationContext = new WSTrustSerializationContext();
// Create RST from the request
RequestSecurityToken request = federationSerializer.CreateRequest(requestMessage, serialisationContext);

// Get RSTR from our STS backend, 
//the thread's principal would not be an IClaimsPrincipal, so create one from the contained identity
IClaimsPrincipal principal = ClaimsPrincipal.CreateFromIdentity(Thread.CurrentPrincipal.Identity);
//issue the RSTR
RequestSecurityTokenResponse response = sts.Issue(principal, request);
// Create Response message from the RSTR
SignInResponseMessage response = new SignInResponseMessage(new Uri(response.ReplyTo), 
                                                           response, 
                                                           federationSerializer, 
                                                           serialisationContext);

response.Write(Page.Response.Output);
Response.Flush();
Response.End();

I’m creating an instance of the STS for each request, but am re-using the sts configuration class (which I keep as an “Application” variable in the STS’ asp.net application).

I’m then using a federation serialiser to create the RST, run this through the STS (providing the principal, “converted” to a ClaimsPrincipal) and then create a SignInResponseMessage out of the RSTR returned by the STS;

Job done – my STS now supports single sign in without the control;

You can already imagine what I needed to do to complete the sign out support- to start with I needed to add an else-if to handle SignOutRequestMessage (as I’ve mentioned – there are other types of requests theoretically possible, but lets not worry about them at the moment) -

else if (message is SignOutRequestMessage)
{

The first thing I would do there, is sign out the user from the STS itslef -

FormsAuthentication.SignOut();

All I needed to do now is add bog standard ASP.net code to generate the required Http Get requests to all my RPs; but to do this I needed to keep track of which RPs a user had visited within the session, so I know which RP’s to sign her out of; to help me achieve that I’ve created the following class to track the user’s visited realms -

public class VisitedRealmsTracker
 {
     private Dictionary<string, string> visitedRealms = new Dictionary<string, string>();

     public void Add(string sessionId, string realm)
     {
         string key = sessionId + "|" + realm;
         lock (visitedRealms)
         {
             if (!visitedRealms.ContainsKey(key))
                 visitedRealms.Add(key, sessionId);
         }
     }

     public IEnumerable<string> GetAllRealmsForSession(string sessionId)
     {
         //find all visited realms for this session and return the second part of they key (after the '|') which would be the realm
         return from visitedRealm 
                in visitedRealms 
                where visitedRealm.Value == sessionId 
                select visitedRealm.Key.Split('|')[1];
     }

     public void ClearUserEntries(string sessionId)
     {
         lock (visitedRealms)
         {
             List<string> keys = 
                 visitedRealms.Where(realm => realm.Value == sessionId)
                              .Select(realm=>realm.Key).ToList();
             foreach (string key in keys)
                 visitedRealms.Remove(key);
         }
     }
 }

I’ve added a member of this type to my STS Configuration class, which – you would remember – I now keep as an application variable, and so I could add a call to Add in the STS code handling the sign in request I showed earlier, which would ensure I’m tracking all the visited realms; the sign out logic could now iterate over the results of the GetAllRealmsForSession and handle the logout requests, simple code to achieve this could be something like -

foreach(string realm in stsConfig.Tracker.GetAllRealmsForSession(Session.SessionID))
{
    //get realm configuration
    ReliantPartyConfigurationElement rpConfig = Configuration.ReliantParties[realm];
    //create an image pointing the at realm's signout url appending the signout and cleanup action
    Image img = new Image();
    img.ImageUrl = rpConfig.SignOutUrl.Trim()+"?wa=wsignoutcleanup1.0";
    Repeater1.Controls.Add(img);
    //add a line break after each realm
    LiteralControl br = new LiteralControl("<br />");
    Repeater1.Controls.Add(br);
}

This sample code simply creates the same images I’ve previously had hard coded in the test page dynamically.

With all the code in place – sign in requests are being processed by code instead of the control, with the code now customised to keep track of visited realms, sign-out requests use this tracked information to dynamically build a page that issues the required Http get request to all the visited RPs to sign the use out of all of them; single sign out achieved. easily.

Geneva Framework and Url case sensitivity- solved?

2009-06-12T11:26:00.000+01:00

I’ve blogged before (somewhat briefly, for a change) about my surprise when I learnt that URLs are [largely theoretically, in my view] case sensitive and the problem that this causes for a Geneva Framework based passive STS implementation.

In that post I mentioned a solution suggested by Peter Korn at the time – setting the path of the cookie to the domain root (‘/’) instead of the application path (including virtual directories), as, unlike the rest of the path, the domain name in a URL is not case sensitive, this works well, and I though it was “case closed”; until recently, when I’ve realised this solution has a very significant drawback - as the cookie, containing the authorisation token from the STS, is stored at the root of the domain, it will be served to every application under that domain, which is taking single-sign-on slightly too far :-)

Following this approach it is not possible allow access to one application and deny it from another (on the same domain) other than through claims processing in the applications themselves, which is a less secure approach from an architecture perspective); clearly not a good solution then…

So – I needed to go back to storing the cookie in the correct path, which would ensure that the STS is re-visited when trying to access a second application (even in the same domain), which – in turn – would mean that the user’s permissions are re-evaluated, before a second, application-specific, token is provided; with that - came back the problem of the URLs being case sensitive.

Thankfully, we’re now on the TAP program for the Geneva Framework, and we’re getting great support by the guys at Redmond (can’t thank them enough!), and after bringing up this issue in a discussion, Shiung Yong suggested another approach to solving this - overriding the GetReturnUrlFromResponse method in the WSFAM.

(Side track: The more I work with the Geneva Framework the more impressed I am with the extensibility options it provides, sure – it’s hard to figure those out on your own if you don’t know about them, and yeah – the resulting solution is often somewhat fragmented, with bits of code in several places, but that’s not much different from many other solutions in this space I suspect – you can see this with many WCF implementations – on the upside, however, if you’re willing to put the sweat, you can do pretty much everything (but yes – the continuum moves from adding a couple lines of code to re-writing the framework :-) )

To understand why and how Shiung’s solution works, consider the following scenario, describing the problem (and here’s where my description is bound to get somewhat confusing) -

Out of the box, the flow of circular redirects, when the URL in the browser is entered in the “wrong” casing, is as follows -

The user types in the RP’s URL, let’s say - all in uppercase, into the browser
As the http request to the RP does not contain an authentication token at this point, the FAM at the RP redirects it to the STS, providing the RP’s ‘realm’ to the STS (the ‘realm’ is configured at the RP and is intended to provide a unique URI to the STS, which it can use to identify the RP, and, for example, be used to load the relevant configuration such as which certificates to use when creating the token); the original URI the user had typed in is also provided through the query string (the ru property in wctx); optionally, and crucially, the RP may also provide the wreply query string parameter, based on its configuration; it is expected that the STS will forward the request, after authentication the user, to this address (but this is not mandated), this will become a key point shortly.
Still at the STS the user authenticates (generally using a login screen), and the STS redirects the request, with a ‘sign in response’ message containing an authentication token back to the RP; as mentioned before it is expected that this would be the address provided by the wreply (and this would be the default behaviour provided by the framework, but this can be easily overridden in the STS’ implementation); for this example, lets assume that the configured value, echoed in the wreply property is set to be in lowercase (remember – the user typed in the URL in the browser in uppercase).
The redirect request contains the set-cookie instructions with the token from the STS and so the browser would set the required cookies in the address the STS redirected to - the lowercase address.
In the step that would follow, the FAM does its sign-in ‘magic’, which concludes by redirecting the request to the URL set in the STS’ response message through the ru field in the query string - this is the URL the user typed into the browser initially, kept by the RP and then the STS - which is all uppercase
At this point, FAM is called again for the new request, attempting to extract the authentication cookie, but as the cookie was stored on the URL the STS redirected to – which was lowercase - and the browser is now using the URL the user typed initially – which is all uppercase - the cookie is not served by the browser, and thus not found in the server code, and the user is being redirected back to the STS as if this was the first call;
As the request arrives to the STS with the uppercase url again, the above would happen again and again in an endless cycle.

Confused? hopefully not too much…but to summarise - out of the box, if the two (the URL configured as the reply to address in the RP, or any other URL the STS uses to redirect back to the RP) and the URL typed into the browser by the user) are not [case-sensitive] identical, the cookie will be set, but subsequently not found when attempted to be read and thus authentication at the RP would continuously fail.

In comes Shiung’s solution -

As long as there’s a convention in the implementation as to the correct form of the URLs (or if drowning in more configuration is acceptable) the FAM could be extended to over come this -

Step 5 above mentions the FAM has some ‘magic’ authentication work with a redirect in the end; the built in implementation uses the ru field to obtain the address to redirect to, but there’s a good extension point there in the form of the GetUrlFromResponse method of the FAM which is called to obtain the url; by overriding this function you can provide whatever logic you wish to control the URL the FAM would redirect the request to after authenticating the request.

Lets say we can agree (as we have) that all reply to addresses will always be configured in lowercase (whilst we can’t control user behaviour, we can control our own configuration), with that agreed we can override the GetUrlFromResponse to always convert the ru value to lowercase before returning it to the bulit in functionality – here’s my version of the method, as suggested by Shiung -

   public class CaseInsensitiveFAM : Microsoft.IdentityModel.Web.WSFederationAuthenticationModule
    {
        protected override string GetReturnUrlFromResponse(System.Web.HttpRequest request)
        {
            string returnUrl = base.GetReturnUrlFromResponse(request);
            return returnUrl.ToLower();
        }
    }

(it is important, of course, to remember to configure the RP to use your custom FAM and not the build-in one -

        <!--<add name="WSFederationAuthenticationModule"
                 type="Microsoft.IdentityModel.Web.WSFederationAuthenticationModule, 
                       Microsoft.IdentityModel, Version=0.6.1.0, Culture=neutral, 
                       PublicKeyToken=31bf3856ad364e35"/>-->
        <add name="CaseInsensitiveFAM" type="CaseInsensitiveFAM, Utilities"/>

)

What had just happened?

By convention, we ensured the RP provided a lowercase reply to address to the STS.

The STS uses this (lowercase) address to forward the request containing the authentication token, and this is where the cookies will be set.

The FAM uses GetUrlFromResponse to retrieve the URL to redirect to, my customised version ensures this would always be lowercase, aligned with the RP configuration

The browser is redirected, again to the lowercase address, this time to receive the cookies set in step 2 which means the request is now authenticated and the user is let in; no more circular redirects!

Of course I’ve implemented a hardcoded rule (always lowercase), but you could use configuration, investigate the http request message or any other logic you’d like…

Some issues remain with that approach (that I can think of) -

If, at this point, the user goes and types the URL in a different casing, as the cookie already exists and the FAM code will not execute again, the user will get redirected to the STS for authentication, but that’s fair enough – I don’t know of any user that would do that in real life..and the result (requiring re-authenticaiton) is quite acceptable

The other thing is that this solution would break should an application be case sensitive (for query string parameters, for example), but we don’t have that problem, and it could be handled by more sophisticated code in the custom FAM, so that’s ok as well.

I suspect this is not the clearest post I’ve ever published (but, unfortunately, probably not the worst), so I can only hope someone will manage to make sense of it and find it useful; I’m pretty sure I’ll need it for future reference; there’s no chance I’m remembering all of this!

Oslo based solution for deploying BizTalk applications – published

2009-05-26T09:35:00.001+01:00

Unfortunately life is a bit hectic for us at the moment, and it’s been a while since I’ve posted anything, or was able to do anything other than family or work.

As part of this general “neglect” I was unable to spend the time required to complete, and publish, my Oslo based solution for deployment BizTalk applications (given the somewhat ugly, but suitably short, name – BTSDF), but as many waited (ok – more than 5), and some kept asking (2), I thought I’d do my best to get something out of the door, so I have.

in Codeplex you can now find an initial version published, which includes – the language definition, the core runtime I have as well as two “executors” in various stages of [non] completeness – my MsBuild generator is already quite useful for simple apps, it generates a set of MSbuild files you can use to deploy the application on any machine as well as the required SDC tasks dlls and targets file; the API executor deploys the app on the current machine, and is quite basic, but a good sample (I think) and a reasonable starting point.

I’m happy to entertain requests for changes/additions, and even more happy to add any one who’s willing and able to put the time as a contributor; Oslo knowledge is optional! :-)

You can download the source, build and run it locally – but you will have to remember to copy the two executors dlls from their folders to the main bin\debug folder as there is no compiled reference to them.

I’ve also uploaded an “Alpha release” which includes the compiled assemblies and the supporting files required.

There’s still more work to do, but it’s getting shape now, and I’m using the MSbuild executor today; it certainly needs a bit more documentation, which I hope to get around to…

I hope this works for you, let me know what you think (the good and the bad), and – if you have some time and will – drop me a line and I’ll add you to the team!

Yossi

RoleClaimsMapper for the Geneva Framework

2009-04-01T08:35:00.001+01:00

Briefly back on my STS work -

Our STS implementation can already replace the authentication implementation of most of our applications; naturally we can’t do that just yet, given that the Geneva-framework has not been released yet, but all of my tests are quite positive so we’re just waiting for the opportunity to start using it.

However, so far, we were not in a position to replace the authorisation mechanism, not easily anyway, and that’s something that was on my list for some time now.

The STS provides a list of claims, which the applications can relatively easily access via code, as many samples show, and this proves very useful; application can investigate various claims about a user and drive their functionality from that.

It does mean, though, that the applications need to change to support this new claims based mode for authorisation, which is not something we can just assume we would be able to do; as a start, we just want to achieve an in-place replacement for our current authorisation logic.

Most of our web apps currently use ASP.net membership and roles and so they extensively use ‘IsInRole’ checks to figure out user authorisation and drive the application behaviour, to start with, we had to hook to that mechanism.

Luckily the Geveva framework has a relatively good support for exactly this need - , out of the box, it would convert any claims of the Microsoft role namespace (‘http://schemas.microsoft.com/ws/2008/06/identity/claims/role’) to roles; so – if a token included a claim of this type with a value of ‘Manager’, a call to HttpContext.Current.User.IsInRole(“Manager”) would return true.

And so I made sure my STS adds any roles with the correct claim type, very easy.

However – this is very Microsoft centric. what about all those claims that come from systems that don’t follow Microsoft’s approach? (how dare they!) ? and what about us wanting to have our own claims, using our own types, some matching roles (while others may not…) -

Well – we needed a way to map any claims to ms-role claims before the Geneva framework does its bit.

As is often the case - Dominick Baier was most helpful in posting on exactly that, and so, following his example, I created my RoleClaimsMapper -

    public class RoleClaimMapper : ClaimsAuthenticationManager
    {
        public override IClaimsPrincipal Authenticate(string endpointUri, IClaimsPrincipal incomingPrincipal)
        {
            //load configuration section for component
            RoleClaimsMapperConfigurationSection config = 
                (RoleClaimsMapperConfigurationSection)ConfigurationManager.GetSection("RoleClaimsMapper");
            //create a collection of claim types and populate from configuratoin
            List<string> claimsToMap = new List<string>(config.RoleClaims.Count);
            foreach (RoleClaimConfigurationElement claimElement in config.RoleClaims)
                claimsToMap.Add(claimElement.ClaimType);
                
            //loop on all identities, we really only expect one, but can easily support multiple.
            foreach (IClaimsIdentity identity in incomingPrincipal.Identities)
            {
                //extract the claims that we need to map (matching the configured list of claims)
                IEnumerable<Claim> roleClaims = 
                    identity.Claims.Where<Claim>(c => claimsToMap.Contains(c.ClaimType));
                //now create a role claim (using the MS role claim type) for each claim found; 
                //need to keep this outside claim loop so we don't modify the collection while iterating
                List<Claim> claimsToAdd = new List<Claim>(roleClaims.Count()); 
                foreach (Claim claim in roleClaims)
                    claimsToAdd.Add(new Claim(Microsoft.IdentityModel.Claims.ClaimTypes.Role, claim.Value,claim.ValueType,"local",claim.Issuer));
                //add new claims to current identity
                identity.Claims.AddRange(claimsToAdd);
            }

            return incomingPrincipal;
        }
    }

I then configured my authentication manager with the framework -

<microsoft.identityModel>
    <claimsAuthenticationManager type="RoleClaimMapper,Identity.Utilities" />

and added my bit of custom configuration

  <RoleClaimsMapper>
    <RoleClaims>
      <add Type="http://someDomain.com/identity/claims/SomeRole"/>
      <add Type="http://someDomain.com/identity/claims/AnotherRole"/>
    </RoleClaims>
  </RoleClaimsMapper>

As you can see this code would take a list of claim types from configuration, and map all claims of these types to roles, adding them to the identity’s claims collection using the required claim type (leaving the original claim intact), and voila – when the app executes it can check the roles, corresponding to the values supplied in my custom claims using -

HttpContext.Current.User.IsInRole(“[custom claim value"]”);

Very nice indeed!

UPDATE: Dominic’s comment on this post (on http://geekswithblogs.net/Connected/archive/2009/04/01/roleclaimsmapper-for-the-geneva-framework.aspx#453535) had indirectly suggested an even cleaner solution; instead of duplicating the claims, I can add the claim types I have as roles to each identity’s RoleClaimTypes collection; this achieves the same result in a much cleaner way, here is the updated function -

        public override Microsoft.IdentityModel.Claims.IClaimsPrincipal Authenticate(string endpointUri, Microsoft.IdentityModel.Claims.IClaimsPrincipal incomingPrincipal)
        {
            //load configuration section for component
            RoleClaimsMapperConfigurationSection config = 
                (RoleClaimsMapperConfigurationSection)ConfigurationManager.GetSection("RoleClaimsMapper");
                
            //loop on all identities, we really only expect one, but can easily support multiple.
            foreach (IClaimsIdentity identity in incomingPrincipal.Identities)
                //for each identity, add all the claim types that are role claim types.
                foreach (RoleClaimConfigurationElement claimElement in config.RoleClaims)
                    identity.RoleClaimTypes.Add(claimElement.ClaimType);

            return incomingPrincipal;
        }

Oslo based solution for deploying BizTalk applications – the runtime

2009-03-29T09:51:00.001+01:00

This is a third post in a series describing my Oslo based solution for deploying BizTalk applications; I’ve used this exercise to play around with ‘M’, but it was important for me to work on a real solution, with real benefits – something I could actually use…in Part I I discussed the concept and presented both the “source code” of my app and the output I was working toward; Part II was all about the MGrammar part of the solution.

In this, third, part I will discuss the last missing piece –the runtime.

Before I start, though, I would say that I did find getting into ‘M’ somewhat confusing at first; and while it’s more than just possible I’m still missing some things , I hope this series could help one or two people in their journey with Oslo – which is, without a doubt, an exciting one!

There are two things, I believe, that contributed to my confusion - the first is the fact the M is really many things, quite different things, actually; from what I hear Microsoft have identified the challenge some of us (me) are having getting a grasp on ‘M’ and are hard at work to bring things [closer] together; hopefully it won’t be long before we know how the converged language looks like, in the mean time one simply has to remember that -

There’s MSchema - which you could use to define models, a bit like xml-schema, or declaring your classes in code or even tables in SQL; I haven’t really touched on MSchema in this series butI might come back to that later.

Then there’s MGraph, which is a way to define instances of things, possibly ones that have been modelled using MSchema, but, as is evident from my little project, not necessarily - MGraph can be very useful even if you don’t have a model- as long as you have your grammar – in comes MGrammar, the third spect of ‘M’, which can be used to define a syntax for your very own [domain-sepcific-]languge for describing things;
A ‘runtime’ could then be used to processes instances described as MGrammar as a result of inputs in your language.

And that is the second thing that really confused me – what is that ‘runtime’? in all the ‘M’ presentations I’ve seen, the ‘runtime’ was merely mentioned and has never received enough “floor space” and yet – an MGrammar without a runtime, in the majority of cases, is, quite useless; you have to have a runtime that would act on your source code; in fact – the runtime would act on the MGraph resulting from your language, which is what makes it all so brilliant, because in a sense, this is where everything comes together – you runtime can work on instances described in your language, on MGraph instances stored in the repository created using MSchema and possibly even ones defined using Quadrant.

The point is that there must be a runtime that understand the model behind your language , can parse its graph and then do whatever you need it to do; and it is your job to build that runtime.

So what have I done for my runtime? here’s a quick overview (reminder: the full source code will find its way shortly onto codeplex) -

My runtime is a console application, one that takes a source code file path as an argument and outputs MSBuild files (and dependencies) that can be used to deploy the application described in the source code onto BizTalk Server.

The first part of my runtime - which I will not bore you with - is about validating the command line arguments; standard stuff.

The second part is about creating the parser for my language, where, thankfully, the Oslo SDK does all of the heavy lifting – it includes a class called DynamicParser which, once created, you can use to parse your source code.

To create the DynamicParser you must first compile your language, and that’s easy enough to do – you start by creating a compiler

MGrammarCompiler compiler = new MGrammarCompiler();

and continue by supplying your grammar

compiler.SourceItems = new SourceItem[] {
        new SourceItem {
            Name="BTSDeploy",
            ContentType = ContentType.Mg,
            TextReader = new StreamReader(GetLanguageDefinition())
        }
    };

(GetLanguageDefinition() is a simple helper method I wrote to get the grammar file embedded as a resource in the exe)

Now you’re ready to compile your language, but to make things manageable you want to provide it with an error reported; the compiler would report any errors to the stream you would provide, I’ve naturally used the console

TextWriterReporter errorReporter = new TextWriterReporter(Console.Out);
if (compiler.Compile(errorReporter) != 0 || errorReporter.HasErrors)
{
    Log("Failed to compile language definition\nSee above for details");
    return null;
}

If the compilation succeeded you are ready to create your parser -

DynamicParser parser = new DynamicParser();
compiler.LoadDynamicParser(parser);

That’s part one of three done.

The next step is to use the dynamic parser to parse your source code, the output of which would be a graph representation of the source; luckily the SDK does virtually all the lifting here as well, and it comes down to one line -

object rootNode = parser.Parse<object>(sourceCodeFileName, null, errorReporter);

Note that the output type is object – which, as you will find out if you try this out, is quite painful– currently all the types used in the Graph are internal, which makes debugging quite difficult (you can’t quite look at any variables you hold in any meaningful way, you have to keep calling methods, as you’ll see next; hopefully this will change one of the next updates to the SDK.

In any case rootNode is now pointing at the root of a graph – a tree like structure you could ‘walk’ to extract the pieces of information you care about in the source code; here you’re expected to use methods like GetLabel, GetSequenceElements and GetSuccessors to reach nodes and their values in the graph and, of course, to do that you need to know exactly how your graph looks like; my first instinct was to look at the PreviewMode pane in intellipad (usually the right most pane when working with MGrammar) as it shows you a representation of the MGraph created for the source code and language used; this worked quite well, but, as I found out, wasn’t the most trivial thing – the two didn’t align completely and I ended up having to resort to trail-and-error to get the parsing logic right.

The reason is that M has a few shortcuts one could take, but the graph you would be working on is the very basic, more verbose format; some information on this is mentioned here.

Then, on a recent visit to Redmond, Dana Kaufman passed on a great tip – if you ‘compile’ your grammar using mg.exe to create the mgx file (basically a ZIP file containing XAML representation of language) and then use mgx.exe on your source file adding a reference to the mgx file you just created, you end up with an ‘M’ file which is exactly the graph your runtime would be working on.end up with; so useful!

So – here are a few example of how I worked the graph – to start with I knew my root node should be a node with the label ‘Application’, so I checked it this way -

string label = graph.GetLabel(rootNode).ToString();
if (label != "Application")

I then knew that the application name would be a child element of the root node, so I extracted it like this

// extract the application's data - this should contain two nodes - the application name and the list of items in the application
List<object> appData = graph.GetSequenceElements(rootNode).ToList<object>();
//first line should be the application name, make sure it is not a node and extract the label
if (!graph.IsNode(appData[0]))
    Contents.AppName = appData[0].ToString();

the second node in the appData collection is where the graph ‘continues’, so to get the list of things that compose my application I needed to walk down that path -

//the second element should be the list of lines
foreach (object section in graph.GetSuccessors(appData[i]))
{
  //each successor would be a category (reference, importing binding, resource, etc), with a list of items
  processSection(graph,section);
}

with processSection start with

string sectionName = graph.GetSequenceLabel(graph.GetSuccessors(section).First()).ToString();
List<object> items = graph.GetSuccessors(graph.GetSuccessors(section).First()).ToList<object>();

Log("Found section '{0}'", sectionName);
switch (sectionName)

I hope that from these few examples you can see what it takes to work the graph – the graphBuilder (which is a somewhat confusing name, as I’m using it to walk the graph, not build it) has all the methods you need to access the various nodes ( but there’s no xpath-like- support), but as all the types are (currently) internal to the MS assembly you’re always working with objects, which is less then ideal.

Again – my full source code is on its way to codeplex, I just want to make sure it’s commented well enough to be well understood, and am struggling with time, but the bottom line is that once you figure out how the graph builder works, learnt how to see your graph visually (using mg.exe and mgx.exe) and got used to the fact that you’re dealing with objects for now, parsing the source code is very easy.

Obviously it is completely down to you what you then do with all the information you’ve extracted from the source code; in my case my runtime is using a plug-in model so the first part is all about using the Oslo SDK to get an instance of a BizTalkDeployment class populated based on the contents of the input file, this class looks like -

    public class BizTalkDeployment
    {
        public string AppName { get; set; }
        public List<object> References { get; set; }
        public List<object> Build { get; set; }
        public List<BizTalkAssembly> BizTalkAssemblies { get; set; }
        public List<object> ImportBindings { get; set; }
        public List<Binding> AddBindings { get; set; }
        public List<Assembly> Resources { get; set; }
    }

I then use late binding and configuration to load a plug in that would take an instance of this class and do the work, be it generation of msbuild scripts, deploying to the local machine using BTSTask or anything else.

On Atomic Scope and Message Publishing

2009-03-23T09:15:00.001Z

A few weeks back I worked on a process that looked something like this -

It was triggered by the scheduled task adapter and then used a SQL send port to call SP to return list of ‘things’.
It needed to split the things in the list to individual records, and to start a new, different, process, through pub/sub (to avoid the binary dependency with the called process), for each ‘thing’.

Fairly simple.

A lot of have been said on the different ways to split messages, I won’t repeat this discussion here; I would just say that initially I used a different approach – I used the SQL adapter in the initial, triggering, receive port and then used a receive pipeline, with an XmlDisassembler component, to split the incoming message so that each record was published individually thus avoiding the need to have a ‘master process’; that back fired though, in my case – I quickly realised I’ll be choking the server with the amount of messages published and needed a way to throttle the execution; I’ve played a bit with host throttling but then came to the conclusion the best approach for me would be to throttle in a process, which is what I’ve done.

And so - to make things interesting, and because I already had it all ready - I decided to use a call to a pipeline from my process to split the message.

The first thing I realised, trying to take that approach, was that I had to change type of the response message received from the SQL port to be XmlDocument (which is an approach I generally dislike – I’m a sucker for strongly-typed-everything) – but my schema was configured as an envelope so that when I call the pipeline from my process it knows how to split it correctly, but, when used in the SQL port BizTalk split the message too early for me – I needed to whole message in the process first, which was no good to me; if , however, I removed the envelope definition from the schema when I would call the pipeline directly from my process it won’t know how to split the message, which is no good either; nor could i have two schemas (BizTalk, as we all know, dones’t like that bit at all, not without even more configuration); XmlDocument it is.

It then came back to me (in the form of a compile time error :-)) that the pipeline variable has to exist in an atomic scope, and so I added one to contain my pipeline variable; I then added the necessary loop with the condition set to the GetNext() method of the pipeline and in each iteration constructed a message using the GetCurrent() method; all standard stuff.

I would then set some context properties to route my message correctly and allow me to correlate the responses (I used a scatter-gather pattern in my master process) and published it to the message box

What I noticed when testing my shiny new process was that all those sub-processes that were meant to start as a result the published messages in my loop were delayed by quite a few minutes (6-8), which seemed completely unreasonable, so I embarked on a troubleshooting exercise which resulting in that big “I should have thought of that!” moment.

While the send shape in my loop successfully completed its act of publishing the message in each iteration, moving my loop to the next message and so on, being in an atomic scope BizTalk would not commit the newly published messages to the message box database, allowing subscriptions to kick in, before the atomic scope would finish; that is to allow it to rollback should something in the atomic scope would fail.
What it meant for me though, was that all the messages were still effectively published at once, which brought me back to square one (or, minus one, actually, considering that the great delay caused my this approach means I’m even worse off from my first debatch-in-pipeline approach).

And so I went back to the old and familiar approach of splitting the messages using xpath in the process, which allowed me to carefully control the publishing rate of messages for my process and throttle them as needed.

ASMX, WCF and enums woes

2009-03-18T10:17:00.001Z

We’ve been slowly migrating our services from asmx to WCF, but as we’re still using BizTalk 2006 with no support for WCF we’ve been exposing endpoints configured for basicHttpBinding and consume them using the SOAP adapter.

Generally speaking things have been going well, although we completely gave up on the idea of moving the services to WCF and NOT have to change the client, until yesterday we’ve stumbled into a serialisation issue –

The SOAP adapter, as part of its work deserialises the request message arriving through the send port into t he web service proxy class it generated, before calling the web service (which would result in the class now being serialised back into xml, which is another story); that deserialisation failed.

The error message was clear enough and indicated it failed to deserialise an enum parameter the service was expecting, and that ran a bell – I posted on exactly that back in September, but after carefully checking and re-checking everything we could swear that our message (which was now suspended) matches perfectly the schema generated by the add web reference wizard; what’s going on then??

After chasing our tail for a short while we brought up reflector to the rescue and found out the cause of our woe is a combination of a difference in behaviour between WCF and ASMX and the use of BizTalk – here are the details –

Consider the following asmx web method –

[WebMethod]
 public string GetDataUsingDataContract(CompositeType.someEnum myEnum)
 {
     return "Hello World";
 }

With CompositeType being

public class CompositeType
 {
   public enum someEnum
   {
      Value1,
      Value2
   }
 }

(..and pretend CompositeType has many more things, but these are irrelevant to this topic)

The definition for myEnum in the WSDL looks like

<s:element minOccurs="1" maxOccurs="1" name="myEnum" type="tns:someEnum" />

Where the type tns:someEnum looks like

<s:simpleType name="someEnum">
    <s:restriction base="s:string">
    <s:enumeration value="Value1" />
    <s:enumeration value="Value2" />
    </s:restriction>
 </s:simpleType>

As a result the definition of the enum in a proxy generated via the add web reference VS 2005 option (which is what BizTalk would use) looks like –

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "2.0.50727.3053")]
 [System.SerializableAttribute()]
 [System.Xml.Serialization.XmlTypeAttribute(Namespace="http://tempuri.org/")]
 public enum someEnum

 {
     Value1,
     Value2,
 }

All makes sense.

Now, let’s look at what WCF does in the same case; consider the following service –

[ServiceContract]
 public interface IService1
 {

   [OperationContract]
    string GetDataUsingDataContract(CompositeType.someEnum myEnum);
 }

 [DataContract]
 public class CompositeType
 {

    public enum someEnum
    {
        Value1,
        Value2
    }
 }

The WSDL generated looks like

<xs:simpleType name="CompositeType.someEnum">
 <xs:restriction base="xs:string">
   <xs:enumeration value="Value1" />
   <xs:enumeration value="Value2" />
 </xs:restriction>
 </xs:simpleType>
 <xs:element name="CompositeType.someEnum" nillable="true" type="tns:CompositeType.someEnum" />

The key difference is that the name of the class containing the enum has made it into the type name for the enum, which never happened in the ASMX version.

As a result the proxy is generated as such -

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Runtime.Serialization", "3.0.0.0")]
 [System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                  Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value2 = 1,
 }

Again – note the name given to the element now contains the class name and, crucially, a dot (‘.’).

On it’s own – nothing to malicious – although it’s another nail in the coffin for the idea that you can substitute web service with WCF service, configured them to use basicHttpBinding and all should be the same (ok – am I the only one still wishing this was possible?)

Enters BizTalk.

When you use the add web reference wizard to add a reference to the WCF service, BizTalk generates all the schemas and proxy for you, which is what you would use to create requests going to the service (and process responses).

Because the WSDL of the WCF service contains the longer name of the enum (with the class name, the dot and the enum name) the .net proxy generated is identical to the one created for the WCF service above; the schema, however, is generated incorrectly!

BizTalk “kindly” decides that having dots in the element name is not a good idea and removes it so the schema generated looks like this –

<xs:schema xmlns:tns="http://schemas.datacontract.org/2004/07/WcfService1" elementFormDefault="qualified" 
          targetNamespace="http://schemas.datacontract.org/2004/07/WcfService1" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element name="CompositeTypesomeEnum" type="tns:CompositeTypesomeEnum" />
 <xs:simpleType name="CompositeTypesomeEnum">
   <xs:restriction base="xs:string">
      <xs:enumeration value="Value1" />
      <xs:enumeration value="Value2" />
   </xs:restriction>
 </xs:simpleType>
</xs:schema>

“CompositeTypesomeEnum”??????

Well, we’ve seen this, and created a message with exactly that element, which – of course – the SOAP adapter failed to deserialise into

[System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                                                    Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value2 = 1,
 }

The solution was fairly simple – we’ve simple change our xsl to put the element name as the .net proxy requires it, and not as the schema describes it, and it all worked well.

Ok, got your message, now let us have xsl re-use!

2009-03-16T08:46:00.001Z

From the Microsoft Knowledgebase -

The <xsl:import> element is used to import an external XSLT file. The <xsl:include> element is used to include an external XSLT file. You cannot use these elements in custom XSLT files that are defined in the Custom XSL Path Grid Properties in a BizTalk project. You cannot do this because a Uniform Resource Identifier (URI) that is defined in an external XSLT file may be from a nonsecure source such as the Internet.

Am I (well, and Ben Gimblett here) the only one who thinks this is a lame excuse? since when MS tries to protect developers from stupidity? and in any case, if they really wanted to do that – wouldn’t they have to prevent us from writing ANY code?

Oslo based solution for deploying BizTalk applications – the grammar

2009-03-14T10:45:00.001Z

A couple of weeks ago I published a post describing my Oslo based deployment framework for BizTalk.

Two parts were missing from that post – the actual MGrammar and the runtime that processes the source code files.

In this post I will go over the grammar I created for the framework; I will try to go over the complete grammar explaining the various steps, this is not intended to be a complete description of MGrammar (not that there’s a chance I could write one), but rather an overview by example; for more information on Oslo and ‘M’ visit the Oslo Dev Centre on MSDN

It was important for me to create a solution that is completely usable, and indeed I have started to use this to generate the build scripts for my application, the price of which is that it might not be the best example code out there, but I hope you will find this useful.

Below is the complete grammar, after which I walk though it step by step; it might be useful to have anther look at the example source code I included in my previous post to better understand what I’m trying to achieve with the syntax -

module Sabra.BizTalk.Deployment 
{ 

    language BTSDeploy 
    { 
        //main syntax is the entry point for the grammar - the first syntax to be parsed 
        syntax Main = app:AppDef 
                           "{" 
                           items:ApplicationItems 
                           "}" => Application[app,valuesof(items)]; 
        //application definition at root of source code 
        syntax AppDef = applicationKW name:ApplicationName => name; 

        //application items supports including all possible items multiple times in any order 
        syntax ApplicationItems = items:(Add | Build | ImportBinding | Comment)* => {valuesof(items)}; 

        //now define the syntax for each item type - 
        //'Build' deinfes a solution or project to build during execution 
        //(due to limitations in our msbuild framework, runtime currently supports solutions only, but language should support both) 
        syntax Build = "build" path:Path";" => Build[path]; 
        //binding to import into application 
        syntax ImportBinding = importKW bindingKW path:Path";" => ImportBinding{Path = path}; 

        //syntax of add further specified different add 'options' 
        syntax Add = addKW add:(Add_Reference | Add_Binding | Add_Assembly | Add_BTS_Assembly) => Add{valuesof(add)}; 
        //each add option is defined next 
        //binding to add as resource to application. must specify environment name 
        syntax Add_Binding = bindingKW path:Path env:MultiWordName";" => Binding[path,env]; 
        //defined a reference to another application, supports providing multiple applications in the same instruction 
        syntax Add_Reference = referenceKW ref1:ApplicationName refs:Add_AdditionalReferences*";" => Reference{ref1,valuesof(refs)}; 
        syntax Add_AdditionalReferences = "," app:ApplicationName => app; 
        //add assembly defines an assembly to be added as a resource to the application 
        syntax Add_Assembly = assemblyKW path:Path details:AssemblyDetails";" => Resource[path,Details{details}]; 
        //add biztalk assembly is similar to assembly, but allows specifiying any contained orchestrations 
        syntax Add_BTS_Assembly = "biztalk" assemblyKW path:Path orch:Orchestrations? details:AssemblyDetails";" => BizTalkAssembly[path,orch,Details{details}]; 
        syntax Orchestrations  = withKW orchestrationsKW "{" type1:ApplicationName types:AdditionalOrchestrations* "}" => Orchestrations{type1,valuesof(types)}; 
        syntax AdditionalOrchestrations = "," type:ApplicationName => type; 
        //assembly details 
        syntax AssemblyDetails = ver:AssemblyVersion+ culture:Culture+ pkt:PublicKeyToken+=>{Version{valuesof(ver)},Culture{valuesof(culture)},PublicKeyToken{valuesof(pkt)}}; 
        token AssemblyVersion = versionKW "=" version:(AnyDigit*"."AnyDigit*"."AnyDigit*"."AnyDigit*)=>version; 
        token Culture = cultureKW "=" culture:Word=>culture; 
        //TODO: token should be 16 chars exactly 
        token PublicKeyToken = publicKeyTokenKW "="pkt:(AnyChar|AnyDigit)*=>pkt;   
        //keywords 
        @{Classification["Keyword"]} 
        token applicationKW = "Application"; 
        @{Classification["Keyword"]} 
        token addKW = "add"; 
        @{Classification["Keyword"]} 
        token bindingKW = "binding"; 
        @{Classification["Keyword"]} 
        token referenceKW = "reference"; 
        @{Classification["Keyword"]} 
        token importKW = "import"; 
        @{Classification["Keyword"]} 
        token buildKW  = "build"; 
        @{Classification["Keyword"]} 
        token assemblyKW = "assembly"; 
        @{Classification["Keyword"]} 
        token biztalkKW = "biztalk"; 
        @{Classification["Keyword"]} 
        token withKW = "with"; 
        @{Classification["Keyword"]} 
        token orchestrationsKW = "orchestrations"; 
        @{Classification["Keyword"]} 
        token versionKW = "version";        
        @{Classification["Keyword"]} 
        token cultureKW = "culture";        
        @{Classification["Keyword"]} 
        token publicKeyTokenKW = "publicKeyToken";        
        //definition of a comment, similar to c# syntax 
        @{Classification["Comment"]} 
        token Comment = "//" CommentLineContent*; 
        token CommentLineContent 
            = ^( 
                 '\u000A' // New Line 
              |  '\u000D' // Carriage Return 
              |  '\u0085' // Next Line 
              |  '\u2028' // Line Separator 
              |  '\u2029' // Paragraph Separator 
            ); 
        //application name must start with a character and then include any character, digit or '.' 
        @{Classification["String"]} 
        token ApplicationName = AnyChar+(AnyChar | AnyDigit | ".")*; 
        //tokens use for definition of a file path 
        token Path = "\""PathRoot?FileSystemName("\\"FileSystemName)*"\""; 
        token PathRoot = AnyChar":\\"; 
        token FileSystemName = (AnyChar | AnyDigit | Space | "-" | "_" | ".")+; 

        //common token definitions 
        token AnyChar = "A".."Z" | "a".."z"; 
        token AnyDigit = "0".."9"; 
        token MultiWordName  = "\""a:(Word | Space)"\"" => a; 
        token Word = (AnyChar | AnyDigit | "-" | "_")+; 

        //the interleave will ensure the language allows whitespace anywhere 
        interleave Whitespace = Tab | LF | CR | Space | Comment; 
        token LF = "\u000A"; 
        token CR = "\u000D"; 
        token Space = "\u0020"; 
        token Tab = "\u0009"; 

       } 
}

I’ve build the grammar top down and this is how I will walk through it -

First (1) I define my module, in a namespace like manner; this is the logical container for my language; I then(4) declare my language and give it a name.

The main constructs in mgrammar are syntax and token; I often heard the guys at Redmond explain that when it comes to languages you can think of syntax as being the sentence and tokens as being the words; I think this is a very clear explanations; there are a few rules relating to them as you can imagine, important ones to remember at this point are that syntaxes can contain other syntaxes as well as tokens (and literals), tokens can only contain other tokens (and literals); also – interleave does not apply to tokens .

The main syntax, and the entry point for any language is Main and you can see mine defined on line 7, and it looks like this -

//main syntax is the entry point for the grammar - the first syntax to be parsed 
syntax Main = app:AppDef 
                   "{" 
                   items:ApplicationItems 
                   "}" => Application[app,valuesof(items)];

‘//’ is used for comments, just like in c#, so the first line will be ignored.

syntax is one of the few keywords that exist in mgrammar, no explanation needed;

Main is the name of the syntax, which allows is to be referred to (used) by other syntaxes; in this case, as I’ve mentioned, ‘Main’ is also the entry point –the syntax the parser will start at; everything else should flow from here.

Now for - app:AppDef, but first – a note - in my mind there are two aspects to creating a language in mgrammar – there is the ‘parsing aspect’ – you define your language so that it describes the rules to parse your source code; and there is the ‘output aspect’ (or ‘production aspect’) – this is where you define the output of your language – in ‘M’ this is mgraph – so that it descries accurately the intent of any source code (and is easy-ish to work with at runtime)

Two are inevitably very mixed in any real-world work with ‘M’ which can be confusing, and today I want to focus on the parsing aspect – firstly because it is the more important one in my view (there’s nothing to work with before you’ve declared a good syntax for your language), and secondly – because I suspect we’re going to see some changes to the production aspect in the near future.

Mainly production aspect ‘stuff’ is defined after the ‘=>’ operator as you can see in my syntax above, so for the time being just try to ignore that; there will be a bit more to ignore as you will see shortly.

AppDef is a name of a syntax declared somewhere else in the language ( line 13); it could also be defined in any imported languages, but I don’t have any, we will look at that in a second; app: is an alias assigned to this syntax which allows for it to be referenced in the production on the right side of the arrow operator; again – for the time being feel free to ignore any aliases, they have no impact on the parsing aspect.

So – my syntax main basically says we’re expecting to have in our source code something that matches the AppDef syntax, then an opening curly bracket then something that matches the ApplicationItems syntax and then closing curly bracket. simple.

Of course next the parser would look at the definition of AppDef and ApplicationItems, and so will we.

AppDef is defined in line 13 as an ApplicationKW followed by ApplicationName; these are defined in lines 51 and 90 respectively; lets look at the ApplicationKW definition-

@{Classification["Keyword"]} 
token applicationKW = "Application";

ApplicationKW itself is a token with a fixed literal 'Application’ – this is a very simple rule to follow, and in fact I could have simply included this literal in the syntax definition and not use this token at all (which is what I have done previously).

The reason I have separated it out to its own token is related to the preceding line in the grammar - the classification attribute allows me to mark this token as a “keyword” for my language, this would tell intellipad (and, presumably, any other editor that would learn how to work with mgrammar), that this token is a keyword and should be displayed as such; in intellipad this means it would be bolded in the editor, as you can see in the image below of my language open in intellipad -

Back to the main syntax’ components - ApplicationName, defined in line 51 states that an application name is composed of AnyChar followed by any number of AnyChar, AnyDigit or the literal ‘.’ with AnyChar and AnyDigit defined in lines 98 and 99.

The ‘+’ sign indicates the syntax or token it follows must exist at least once; the ‘*’ sign indicates the syntax or token it follows can exist 0 or more times.

So – we have the definition of our application name, now lets look at what ApplicationItems says -

syntax ApplicationItems = items:(Add | Build | ImportBinding | Comment)* => {valuesof(items)};

This syntax tells the parser that an application can have any number of Add, Build, ImportBinding or Comment in any order.

Moving on we’ll look briefly at how ImportBinding looks like -

syntax ImportBinding = importKW bindingKW path:Path";" => ImportBinding{Path = path};

The importKW (which is the literal ‘import’, look it up!) followed by the bindingKW (‘binding’) and the syntax for Path.

I could have combined both literals import and binding to a single token and mark that as a keyword, but there are two benefits to splitting them up- firstly, by having two tokens I can have as many whitespaces as I want between them, which I think is what developers generally expect, and, secondly – the 'binding’ keyword is re-used for the add binding syntax I’ll describe shortly.

I’ll skip the Path definition, you can follow it yourself if you wish to; so next we can look at another item in the application items list – Add:

syntax Add = addKW add:(Add_Reference | Add_Binding | Add_Assembly | Add_BTS_Assembly) => Add{valuesof(add)};

The Add syntax starts with the addKW (‘add’) followed by one of the syntaxes for adding a reference, adding a binding, adding an assembly or adding a BizTalk assembly, but it only allows one; the add keyword (and therefore the entire add syntax) must be repeated as a whole to add multiple items to the application, as is suggested by the ApplicationItems syntax.

Lets look at a couple of these items; first – the syntax for add binding -

syntax Add_Binding = bindingKW path:Path env:MultiWordName";" => Binding[path,env];

Here you can see the binding keyword being reused, as does the Path syntax; I’m then allowing a multi-word-name (which is essentially a string contained in double quotes) as the environment name for the added binding.

Quite simple, right? that’s the thing I love about mgrammar. let’s look at one more syntax -

syntax Add_BTS_Assembly = "biztalk" assemblyKW path:Path orch:Orchestrations? details:AssemblyDetails";" => BizTalkAssembly[path,orch,Details{details}]; 
syntax Orchestrations  = withKW orchestrationsKW "{" type1:ApplicationName types:AdditionalOrchestrations* "}" => Orchestrations{type1,valuesof(types)}; 
syntax AdditionalOrchestrations = "," type:ApplicationName => type;

The Add_BTS_Assembly syntax should be very clear, the only thing I haven’t mentioned so far is the ? sign which indicates 1 or 0 appearances of the syntax/token it follows, I use this to allow a BizTalk assembly to optionally describe the orchestrations it contains so that, potentially, any instances of these could be terminated when undeploying the application.

The Orchestrations syntax, if exists, requires at least one orchestration to be specified (I’m reusing the ApplicationName token as the orchestration name) but allows additional orchestratiosn to be specified as well; I’ve used the same approach for the add reference syntax.

I hope this makes sense, and that it gives you a glimpse into a practical use of mgrammar, I am certainly excited about this stuff.

Soon I hope to post about the last missing piece of the puzzle – the runtime that uses the language definition to parse, and then execute, any source code provided; after that the whole thing is likely to find a spot on CodePlex, bare with me a little bit longer….

MSBuild support in BizTalk 2009

2009-03-08T16:49:00.001Z

In my notes on BizTalk 2009 post I have highlighted the fact that now, as part of the BizTalk installation, you can select the “Project Build Components” which allow you to build BizTalk projects without Visual Studio or BizTalk installed on the build machine.

This is very cool feature of the product, but I have to admit that I have missed something quite important; luckily - Mikael Hakansson hadn’t – the provided tasks do not support deployment of BizTalk projects, but only build (see Mikael’s article here).

What this means is that, while you can use the tasks deployed through the installation to verify that your BizTalk projects build correctly (as part of continuous integration setup, for example), you have no way of verifying that they can be deployed or, more importantly, run any automated tests;

The SDC tasks, with their support for BizTalk deployment, can come in very handy here, and you can certainly use them to augment the built in build support with deployment capabilities, but they add more dependencies to the plate, which has to be planned for.

Of course, attending the MVP summit in Redmond last week we had a great chance of raising this (and various other) subjects with the product group, and we certainly have not let that opportunity go missed, now we just need to wait and see if they “pick up the glove”.

Oslo based solution for deploying BizTalk applications

2009-03-04T00:02:00.001Z

Since PDC I’ve been working on and off on an “Oslo” based solution for deploying a BizTalk application; unfortunately I couldn’t get a good chunk of time to play with this, so it’s been dragging a bit, but I’m getting close, so here are some details -

I’m a big advocate of automated builds; it’s a topic that probably deserves a post of its own, so I won’t get started on this here, but the idea is that one must have a way to be confident that, when its time to [re-]deploy the app, it will get deployed successfully time after time, without a hitch;

From my experience, deploying a BizTalk application of a decent size is often not trivial, and often it does not go smoothly if done manually (things are being missed, done out of order, wrong versions being picked up, etc), which does not help to boost the confidence in the solution (or BizTalk as a whole) in the organisation; automating the process can save a lot of time (can be done unattended, while in a meeting, out for lunch or overnight), save a lot of head scratching, boost the confidence in the solution and set the ground for proper automated builds, auto testing etc.

To that extent, I have previously built an MSBuild based framework for deploying BizTalk applications.

While I’m sure it does not suit all purposes as it was developed to support those scenarios I had, I think it’s quite comprehensive and had served me well over the past couple of years.

It allows one to provide an MSBuild-formatted file with some parameters and lists and, using the pre-created framework, would do things like remove the existing application (after terminating current instances and un-enlisting services), build the new solution(s), deploy assemblies to BizTalk, add/import binding files, start ports etc.

This is working great for us, and we’ve been using it extensively for quite a while now, but there’s one major downside – it requires one to maintain those MSBuild scripts.

Now - MSBuild is cool, I do like it very much, but as it’s generic – it does not speak in BizTalk terms, and as it’s XML – it’s quite verbose, and I wanted an easier format for all of us to work with.

So – inspired after seeing Oslo and even more driven after visiting PDC - I’ve decided to come up with an MGrammar based language for describing a BizTalk application to be deployed.

So far I’ve come up with “version 1.0” of my grammar that allows to describe the major artifacts in a BizTalk application and a basic runtime to process source files written in this language, the language allows you to describe things like -

References to other applications
Solutions and projects to build
BizTalk assemblies to deploy
Orchestrations contained in those (these can then be terminated before attempting to remove the application prior to deployment)
.net assemblies to deploy
Binding file to import
Binding files to add (and the name of the environment to attach to those)

As Oslo hasn’t RTM-ed yet, I can’t quite rely on it yet, and so I cannot use it in production in any shape or form, or can I?
I found a good middle ground for us which allows us to gain from the benefits I’m hoping to get by using my language, while not exposing ourselves too much to the risks of using early technology –

At the moment the runtime I’ve built for this is used to generate the MSBuild scripts we’re already using out of the source files; in this way - if Oslo disappears tomorrow, or significantly changes (not that I think that’s going to be the case), we’re safe – we still have the MSBuild scripts as “checked-in artifacts” and so we have lost nothing.

So – how does an instance of my language looks like? here’s an example:

Application MyApp
{
    //add references to other applications
    add reference SomeOtherApp;
    add reference AndAnother,andAThird;
    
    //build the solution
    build "SomeFolder\SomeOtherFolder\SomeSolution.sln";
    build "SomeFolder\SomeOtherFolder\SomeProject.btproj";
    
    //add required biztalk assemblies
    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Schemas.dll"
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;

    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Maps.dll"
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;

    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Orchestrations.dll" 
    with orchestrations
        {
            MyApp.SomeOrchestration,
            MyApp.AnottherOrchestration,
            MyApp.AndAnotherOrchestration
        }        
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;
    
    //and .net helpers assembly
    add assembly "SomeFolder\SomeOtherFolder\bin\release\HRG.FareWatch.Hotel.Helpers.dll" 
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;
    //import dev bindings
    import binding "SomeFolder\SomeOtherFolder\Bindings.xml";

    //add various environment's bindings.
    add binding "SomeFolder\SomeOtherFolder\Bindings.Dev.xml" "development";
    add binding "SomeFolder\SomeOtherFolder\Bindings.Stage.xml" "stage";
    add binding "SomeFolder\SomeOtherFolder\Bindings.Prod.xml" "production";
}

as I’ve said – I already have a runtime that generates the MSBuild scripts required to deploy these - the runtime outputs several files to the temp folder -

The “framework” MSBuild script that contains all the targets I’m using

The Microsoft.Sdc.Tasks.BizTalk.dll and the Microsoft.Sdc.Tasks.dll which contains many useful custom tasks.

The relevant import file for the SDC tasks

The generated MSBuild file that contains all the various artifacts based on the source code

That latter file, which is the equivalent of my source code, and what we had to maintain so far, looks like -

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="DeployAll">
  <Import Project="Deploy.Framework" />
  <ItemGroup>
    <MyApp Include="SomeOtherApp" />
    <MyApp Include="AndAnother" />
    <MyApp Include="andAThird" />
  </ItemGroup>
  <ItemGroup>
    <Bindings Include="General">
      <path>SomeFolder\SomeOtherFolder\Bindings.xml</path>
    </Bindings>
  </ItemGroup>
  <ItemGroup>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Dev.xml"</path>
      <environment>development</environment>
    </EnvironmentBindings>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Stage.xml"</path>
      <environment>stage</environment>
    </EnvironmentBindings>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Prod.xml"</path>
      <environment>production</environment>
    </EnvironmentBindings>
  </ItemGroup>
  <ItemGroup>
    <ExternNetAssembly Include="HRG.FareWatch.Hotel.Helpers">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\release</path>
    </ExternNetAssembly>
  </ItemGroup>
  <ItemGroup>
    <Solution Include="SomeSolution">
      <path>SomeFolder\SomeOtherFolder</path>
    </Solution>
    <Solution Include="SomeProject">
      <path>SomeFolder\SomeOtherFolder</path>
    </Solution>
  </ItemGroup>
  <ItemGroup>
    <BTSProject Include="Schemas">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
    <BTSProject Include="Maps">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
    <BTSProject Include="Orchestrations">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
  </ItemGroup>
  <PropertyGroup>
    <BTApplicationName>MyApp</BTApplicationName>
    <!-- Set for a remote deployment -->
    <!-- Deploying BizTalk Server name - leave blank if local-->
    <!-- not currently supported by runtime-->
    <BTServerName>
    </BTServerName>
    <!-- Deploying BizTalk Server database - leave blank if BizTalkMsgBoxDb-->
    <BTServerDatabase>
    </BTServerDatabase>
    <!-- Deploying BizTalk Server SQL user name - leave blank if local-->
    <BTServerUserName>
    </BTServerUserName>
    <!-- Deploying BizTalk Server SQL password - leave blank if local-->
    <BTServerPassword>
    </BTServerPassword>
  </PropertyGroup>
</Project>

Would you agree that the former source code is easier to maintain?

note: one thing you would notice is that none of the paths contains a root; I assume that this would be used by different developers/IT pros which may have the code in a different path; however, as the assumption is that the code will come from source control, my framework script expects you to provide the root path to your source code and assumes the specified paths start there.

So – what’s next?

From a coding perspective - I plan to add support in my runtime to perform the actual deployment; this is I would like it to do once Oslo is in production so I’ll add it as an option through a command line switch.

This option would tell the runtime to deploy the artifacts to a specified BizTalk server using BTSTask or the Object model instead of generated the MSBuild script.

I also want to make some modification to my language definition to make the MGraph produced cleaner (and better geared towards using the repository and Quadrant at a later stage) as well as, obviously, add support for more features, after which I plan to publish both my language and the runtime somewhere (here, codeplex, DevCente..we’ll see)

I’ll post some more notes on all of these here in the near future hopefully…

Serialisation, mixed content and string[]

2009-02-21T06:57:00.001Z

When you generate a class out of a schema with an element configured to allow mixed content (child attributes and elements as well as text), you should expect the corresponding generated field type to be a string array;

So - if you have a schema that looks like this

<?xml version="1.0" encoding="utf-8"?>
<xs:schema targetNamespace="http://tempuri.org/XMLSchema.xsd" elementFormDefault="qualified" xmlns="http://tempuri.org/XMLSchema.xsd" xmlns:mstns="http://tempuri.org/XMLSchema.xsd"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="SomeElement">
<xs:complexType mixed="true">
<xs:sequence>
<xs:element name="Child1" type="xs:string"/>
<xs:element name="Child2" type="xs:string"/>
<xs:element name="Child3" type="xs:string"/>
</xs:sequence>
<xs:attribute name="SomeAttribute" type="xs:string"/>
</xs:complexType>
</xs:element>
</xs:schema>

(not ‘SomeElement’ being a complex type allowing mixed content)

The fields in the generated class would look like

public partial class SomeElement {
 
 private string child1Field;
 
 private string child2Field;
 
 private string child3Field;
 
 private string[] textField;
 
 private string someAttributeField;
.
.
.

The reason is that an XML corresponding to the schema might look like this –

<SomeElement xmlns="http://tempuri.org/XMLSchema.xsd" SomeAttribute="someAttributeValue">
 Some free text
 <Child1>Child1 text</Child1>
 Some more free text
 <Child2>Child2 text</Child2>
 yet some more free text
 <Child3>Child3 text</Child3>
</SomeElement>

And so by using a string array to hold the text the deserialiser can keep string portion separately.

Initially, I thought, this allows the structure to represent the original xml accurately, but this is not exactly the case – you would still not know for certain where each string portion existed, especially if in the source XML you get a few elements that don’t have text between them, which , I suspect, is why when I serialise the instance back to xml I actually get –

<?xml version="1.0"?>
<SomeElement xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" SomeAttribute="someAttributeValue" xmlns="http://tempuri.org/XMLSchema.xsd">
 <Child1>Child1 text</Child1>
 <Child2>Child2 text</Child2>
 <Child3>Child3 text</Child3>
 Some free text
 
 Some more free text
 
 yet some more free text
 </SomeElement>

Now, I don’t particularly like this sort of xml, and shy away from mixed content; I don’t believe that xml snippets like my samples above are useful, specifically I don’t think that mixing elements and text is particularly nice.

However, consider an element with an attribute and some text – the following is quite reasonable I think, and yet requires mixed content -

Supporting both active and passive scenarios in my STS

2009-02-15T11:04:00.001Z

In a comment on a previous blog post Travis Spencer asked

Can you explain more about how you implemented an STS that supports both active and passive scenarios?

So here’s how -

To start with – I’ve implemented my STS class with all the logic I needed; this was done as a class library with several classes – my STS implementation, my STS-Configuration class, an STS service factory, my custom WindowsUserNameSecurityTokenHandler implementation and all the classes I needed to support my custom configuration section.

Then, in order to support an active scenario, I’ve created a WCF service and, through the SVC file, I’ve configured it to use my STS service factory class -

<%@ ServiceHost Language="C#" Debug="true" Service="<My STS configuration class>" factory="<My STS Factory class>"%>

I’ve then configured the web.config of the wcf service to support my scenario – that included all the relevant binding configuration I needed, the Geneva framework related configuration (microsoft.IdentityModel) as well as any custom configuration my STS uses.

The passive scenario can seem a little bit more confusing -

Obviously I’ve started by creating an asp.net web application; this application basically has two web pages (admittedly I’m simplifying things a little bit for clarity) – default.aspx and login.aspx

Using standard asp.net forms authentication the web site is configured to redirect all unauthenticated users to the Login.aspx page, which in turns has a pretty standard login implementation using my custom username validator logic and the framework’s RedirectFromLoginPage function to set the local forms authentication cookie.

All my web-based reliant parties redirect the user to the default.aspx page; forms-auth then redirects again to login.aspx for authentication and then, once authenticated, the user is redirected back to default.aspx; on this page I’ve simply put the FederatedPassiveTokenService control provided with the geneva framework configured to use my STS configuration class as the service; this takes care of calling the STS and posting the token back to the RP

I hope that makes sense…do let me know if it does not!

VS crashes? check your colour depth setting!

2009-02-07T17:34:00.001Z

Over the last few months we've been seeing quite a few Visual Studio crashes when working with the orchestration designer.

The scenarios vary slightly between machines - some would crash on build, others simply when opening an orchestration in the designer; some would crash very frequently, some only sporadically; some would crash even when opening the solution immediately after a reboot, others would work fine for a while and then crash; the bottom line was that we were having lots of crashes, and we were losing on developer productivity and gaining tons of frustration.

Because the problems did not happen in a very consistent manner, and because there were different manifestations we simply had no clue what might be the cause; we've talked to a few people and searched the web; we’ve followed almost every tip we've received (make sure no documents are open when you build, collapse all shapes in an orchestration before saving it, etc.); some seem to have helped a little bit, but the crashes kept happening.

Sometimes Visual Studio would show an error on the screen – something like -

But in other times, it simply crashed without a warning.

Eventually we’ve contacted Microsoft support, and analysing dumps we’ve created they have found the following error -

Exception type: System.InvalidOperationException
Message: BufferedGraphicsContext cannot be disposed of because a buffer operation is currently in progress.
StackTrace (generated):
System_Drawing_ni!System.Drawing.BufferedGraphicsContext.Dispose(Boolean)
System_Drawing_ni!System.Drawing.BufferedGraphicsContext.Dispose()
System_Drawing_ni!System.Drawing.BufferedGraphicsContext.AllocBufferInTempManager(System.Drawing.Graphics, IntPtr, System.Drawing.Rectangle)
System_Drawing_ni!System.Drawing.BufferedGraphicsContext.Allocate(IntPtr, System.Drawing.Rectangle)
System_Windows_Forms_ni!System.Windows.Forms.Control.WmPaint(System.Windows.Forms.Message ByRef)
System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr, Int32, IntPtr, IntPtr)

From this it was clear that the issue is related to GDI+ when VS redraws the orchestration and MS have confirmed they have seen this issue elsewhere (outside the BizTalk realm), but we were told that this problem was “aggravated by the fact that BizTalk creates very large device-independent bitmaps and uses the same colour depth than the desktop.”

Unfortunately, according to Microsoft, there are many contributing factors to this issue, and it is virtually impossible to isolate the cause(s) in our case, but two recommendations were made -

Use a video card with more memory

Lower colour depth setting from 32 bit to 16 bit in the display settings in windows

Split large orchestrations to smaller ones

That last recommendation would have been our last resort – not only it would require a significant re-factor-re-test effort as our solution is quite big, it would seriously decrease our already too long deployment time.

We were quite happy to follow the first recommendation, but obviously trying the second one was pretty effortless we went with that first and what do you know – it worked!

Machines which have changed the colour depth setting to 16 bit stopped having those frequent crashes;

It must be said we’re still experiencing some crashes, so there are other issues as well, but these are more specific, and are another story altogether, one to be told once the end is known!

Specifying the user for a service to run as

2009-01-25T08:08:00.001Z

Mostly a reminder for myself, but hopefully useful to somebody else -

Often it is important to specify a specific user for a service to run as; it appears the setup is completely different when using IIS 5.1 or 6 (and higher).

When using IIS 5.1

Set the anonymous user on the virtual directory to the user you want to run as.
Disable any other authentication method on the vdir
In the web.config turn impersonation ON (<identity impersonate="true" /> under System.web.)
Under System.serviceModel add <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>

To the service class add [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]

When using IIS 6.0

In the web.config turn impersonation OFF(<identity impersonate="false" /> under System.web.)
Set the required identity in the app pool your virtual directory runs under.

(which one do you prefer? ;-))

Publishing a service with federated identity from BizTalk Server [2009] – Part II

2009-01-13T09:46:00.001Z

In my previous blog post I’ve described how I consumed a service that uses ws2007FederationHttpBinding from BizTalk Server; my next task was to expose an orchestration as a WCF service that uses this binding.

In that post I’ve described what I think is a bug in BizTalk R2/2009 which prevents me from setting the issuer configuration through the UI.

When consuming such a service this configuration exists in the send port, and I’ve managed to get enough time to manually edit a BizTalk bindings file to import in order to set the required configuration I can’t set through the UI; I have tried to do the same for the receive location, but the binding file's format is different and my few attempts before my time ran out did not work.

I have submitted this buy to Microsoft, and hopefully they’ll provide a fix/some information about a possible work around; I suspect my best bet is to use the explorer OM to configure the receive location, but I just can’t get the time to do it now, so – I’m afraid this post is mostly a place holder, and I really hope I’d be able to come and update it with some more details soon.

Calling a service with federated identity from BizTalk Server [2009] – Part I

2009-01-13T09:38:00.001Z

Finally I’ve reached the point where I’m ready to hook up BizTalk to my STS implementation to participate in a federated identity scenario.

My goal is to confirm two scenarios -

1. Being able to call from a BizTalk process a service that uses the ws2007FederationHttpBinding (and requires that the caller provide a token issued by a specific STS)

2. Being able to expose a service in BizTalk that would use the ws2007FederationHttpBinding requiring the caller to provide such token.

If you followed my previous posts you would probably know that I already have a fairly extensive federated identity scenario – I have a custom STS built using the Geneva Framework, a few web sites and a test WCF service all use the STS for their authentication (and, indirectly, authorisation), so it is just a case of hooking BizTalk to all of that, and to start with I decided to try and consume that test service from a BizTalk process.

I’m using BizTalk 2009 Beta to keep things interesting, but I’m pretty sure it would all be exactly the same for R2.

The beginning was easy – in my BizTalk project I’ve used the “Consume WCF Service” wizard from the “Add Generated Items” menu to generate all the artifacts needed to consume my service; the generation went very smoothly and soon enough I had an orchestration that could receive a dummy trigger message, create the service request, receive the service response and deliver it to a send port for me to check it out.

I’ve deployed the project and then imported the binding file generated by the wizard to create the WCF send port.

Checking the port configuration, though, I found a few issues -

The port uses the WCF-Custom adapter, which was correctly configured with the service’s endpoint and the ws2007FederationHttpBinding; however – my STS endpoint is configured to use the ws2007HttpBinding and requires UserName credentials, but these settings were not copied over to the client configuration from the service's contract, the client configuration should have looked like -

(with corresponding Ws2007HttpBinding named stsBinding with all the relevant configuration), but the binding attribute and the bindingConfiguration attribute were missing.

It is worth noting here that this problem is by no means BizTalk specific – exactly the same thing (missing configuration) happens when I add a reference to my service from a ny standard .net app.

Also worth noting that in my case, and I don’t know how representing this is, this resulted in WCF trying to open the CardSpace card selector when I tried running the process, resulting in an error - “The channel is configured to use interactive initializer 'System.ServiceModel.Security.InfocardInteractiveChannelInitializer', but the channel was Opened without calling DisplayInitializationUI. Call DisplayInitializationUI before calling Open or other methods on this channel.”

Easy enough to fix, I thought, as I opened the adapter’s configuration for the send port finding that indeed there are the equivalent fields in the UI (the image shows them AFTER I have typed in the values I needed) -

However, after adding the missing two values and confirming the changes I kept getting the same error; it only took two minutes of head scratching to realise what was going on – opening the send port configuration again showed that the values I’ve carefully typed in have not been persisted, and are now gone again; this must be a bug, which I’ve since confirmed exists in BizTalk 2006 R2 as well;in fact – even the issuer address, which was previously there, was removed from the configuration when trying to apply my additions.

Luckily importing these settings through a bindings file does work (as was evident from the fact that the issuers address did exist initially), and so I’ve edited my bindings file manually to include the missing settings (as well as the issuer identity setting which I later found out was missing for my scenario) and imported it using the admin console; checking the send port configuration I could now see my values for the “binding” and “bindingConfiguration” properties.

This raised another question, though – how do I configure this binding? ws2007FederationHttpBinding is somewhat “special” in the sense that there’s effectively an endpoint inside an endpoint:

Using this binding you have your service’s endpoint and it’s binding configuration; then the message security element of the ws2007FederationHttpBinding has the issuer element which is effectively the endpoint of the STS with the address, the binding and the binding configuration (there is no need for a contract attribute as the contract is known – it is the wsTrust contract); this means that I now need to have another set of binding configuration in my WCF configuration – for the STS – which in my case uses ws2007HttpBinding, but there isn’t a way to add another binding through the send port configuration dialog directly.

Initially I tried to add the relevant configuration to the BizTalk configuration file (BTSNTSVC.exe.config), but it didn’t work – and I can’t quite explain why yet.; what did work, which is good enough for me for now, was to put that exact same configuration in the machine.config; no idea why, but it works.

Another options, described in the documentation is to use the Import option or the ExplorerOM API.

And so – once I’ve done that, when I ran my process it called the send port going to my service, but the WCF adpater contacted my custom STS first, retrieved a security token from it, and then called my test service passing this token, which meant the service was allowed to execute.

I’m impressed!

There is one, fairly big, inefficiency here now, though – BizTalk does not appear to be caching service proxies, which means it will not cache anywhere the issued token, which in turn means a call to the STS must precede every call to my protected service; not ideal.

Luckily - one of the WCF samples provided by MS demonstrates how to create a behaviour that would cache such tokens locally; at some point (probably not right now, unfortunately) I will have to see how well this plugs into BizTalk (if at all), but I suspect it’ll work well; find that sample here.

Wish list: optional parameters when using call/start orchestration

2008-12-29T10:35:00.001Z

I wasn’t sure how bit problem this was, and in any case thought that optional parameters are a VB thing, but I’ve been proven wrong, and not only because optional parameters are soon to become a c# thing :-)

When using call/start orchestration we increasingly, as our solution “matures”, find that we need to add optional parameters to our processes..

In c# I was trained that function overloading is the correct way to implement this, and although I can see the benefit of the slim optional parameters approach I do think that it is much clearer; we’re talking BizTalk though, and you can’t really overload a function, can you? :-)

So we [can] do several things -

When dealing with .net types we can simply break the interface, add the parameter, change all the calling processes to supply one, and – if they don’t actually need to supply a value for this new parameter make them pass one anyway (an empty string, or a 0, or even –1); not elegant to say the least, and quite time consuming which means upsetting both the architect and the project manager.

Same approach works quite well when dealing with optional messages as well, only that now constructing that unnecessary extra parameter, in the same of a content-less message, is even harder (see my post on creating messages from scratch).

Another option, valid for .net types only, is to have a <something>Parameters class which would include all the parameters a process takes as properties; this would allow you to add as many properties as you need, without forcing callers to specify them; there are downsides of course – it is less readable and intuitive in my view, there’s a bit more hassle for the caller (create instance of parameters class, assign members) and now it’s harder to force those non-optional parameters (can be done via the parameter class’ constructor)

But there’s not equivalent option for optional message parameters.

Last, but not least, there’s the closest thing to overloading – you could always create a second process, with the extended set of parameters and have the original process forward the original call while providing this empty parameter, effectively saving the hassle from the original caller.

This is really how a function overloading solution would look like only that a process is not a function; it has a lot more impact in term of compile time, configuration required, deployment so the overall maintenance cost of such an approach is quite enormous for any decent sized project.

So – I can no longer ignore this problem and have to agree – allowing optional parameters with default values, for orchestration is quite important.

2011? :-)

Never thought Url would be case sensitive

2008-12-16T14:27:00.001Z

Working on my Geneva Framework based STS scenario I’ve stumbled into a very weird and annoying case where by if the user typed a Url in the wrong case (compared to the case of the V-Dir) the browser would enter a circular redirect between the STS and the RP.

I’ve started a forum thread, which you can find here, that got an answered by Peter Kron from MS through which I’ve learnt that the path portion of a cookie is case sensitive; you can find this in this RFC spec as well (read 3.3.3) -

…the old and new Domain attribute values compare equal, using a case-insensitive string-compare; and, the old and new Path attribute values string-compare equal (case-sensitive). …

I don’t know if that’s just me, but I find this really surprising as, as a web user, I was never “trained” to tread urls as case sensitive, but it appears that, according to the spec, any personalisation stored for a particular path might be lost if I enter the wrong url?

In the STS scenario case this would mean potentially me having to login again, although I have already logged in on the STS.

Peter suggest to store the cookie against the domain, which is not case sensitive, and is good enough for me (for now?), but I don’t know if that’s realistic for all scenarios…..

Notes on BizTalk 2009

2008-12-11T17:47:00.001Z

earlier this week Microsoft announced the release of the beta version of BizTalk 2009.

I’m sure detailed posts of various bits will follow soon, but for now I thought I’d list a few points I’ve picked up (in no particular order)-

BizTalk projects are now “first class citizens” of Visual Studio [2008]; in practice it seems they are really “special” c# projects.

This means quite a lot really, to start with, for the most part they look and feel like c# projects (in the beta build the icon for the project is even the c# icon), but this extends to the property pages, existence of assemblyinfo.cs and the project file structure (which is now pretty much standard msbuild file)

It also means that you could add any c# type to a BizTalk project although in the current build you can’t add a new class using the menu, you are always redirected to the template selector which only shows BizTalk types, so –even if I pick the Add new class option

I get the following dialog -

Not sure when this will be fixed, but the workaround is simply (yet annoying) – create the type elsewhere and use the “Add Existing Item” option.

This will allow you, for example, to include any helper classes you use from your orchestration with the process itself.

Another point to notice is that this is only true for C# types. you cannot add VB code files to a BizTalk project.

Also – as the project is now a standard msbuild project integration with TFS is tighter.

interesting to see in the installation process the following entry -

Basically installing this on a computer that has msbuild allows you to build BizTalk projects without having BizTalk or VisualStudio installed; using a build server for BizTalk is so much easier now!

The build process is completely incremental – each BizTalk artifact is a c# class, and so when building a project only those classes that have changed will be built, which should shorten the time it takes to build a BizTalk project.

Another fantastic addition is more “HAT” features in the admin console, but as Randal already blogged about this one in details I’ll point you there

Upgrading seems to be a breeze and both 2006 and 2006 R2 are supported. this can be done by opening any existing BizTalk project in VS 2009 and the migration wizard will pick it up and migrate it for you. as expected.

you can open a solution with multiple projects and they will all be upgraded and you can use devenv /upgrade with both projects and solutions to do that from script?

BizTalk 2009 supports debugging maps – this is done by selecting the “Debug Map” from any BTM file context menu -

This would use the test input instance and test output instance properties, and would basically open the XSL generated for this map and break at the root template

Worth noting that currently this does not support maps with multiple inputs or ones using extension objects.

I’m sure there’s a lot more to discover, but I have a 4 day old baby at home – so back to those nappies for me I’m afraid.

Geneva-based passive STS and .net 2.0 web applications

2008-12-04T09:25:00.001Z

Over the last few weeks I’ve been working on implementing a Geneva Framework based STS that supports both active and passive scenarios.

This is progressing very well and I already have a fairly solid PoC running for both scenarios.

Generally, to make any web site participate in the federated identity “dance” all it takes is some configuration on the application’s web.config (separate post coming shortly), but up until today I have only done so for web applications developed as a .net 3.5 project.

Today I have created a bog standard .net 2.0 web application project (on VS 2008, though, not that it should matter) and applied exactly the same configuration as I did on the other web sites.

when I ran this, not so surprisingly, everything just worked. isn’t it great?!

Of course – considering that everything is implemented as HttpModules – this should not be very surprising, but still.

It is important to remember, though, that .net 3.5, as well as the Geneva Framework, must be installed on the machine running the web site for this to work of course.

What I hadn’t appreciated about schema versioning in BizTalk Server

2008-12-03T21:07:00.001Z

Most people know that when processing a message through the XmlDisassembler, if not explicitly told which schema to use through configuration, the disassembler would try to resolve the correct schema based on the message’s root node and namespace.

Most would also know, usually through the experience of getting it wrong so many times first, that if more than one assembly contains the same combination of root node and namespace for a schema, the receive pipeline, containing the disassembler, would fail with the error “Cannot locate document specification because multiple schemas matched the message type “<your message type here>”.”

What some miss (ehm, ehm) is that this is not true if the two schemas exist in two versions of the same assembly.

So – if you have an assembly called MySchemas.dll, version 1.0.0.0, …. which contains schema SomeRootNode#SomeNamespace and you have MySchemas2.dll, Version…… which contains schema SomeRootNode#SomeNamespace (same one) – BizTalk would fail.

If, however, you have the assembly MySchemas.dll, version 1.0.0.0, …. which contains schema SomeRootNode#SomeNamespace and then create MySchemas.dll, version 2.0.0.0, …. which contains schema SomeRootNode#SomeNamespace BizTalk would quite happily ignore the former and use the latest version available.

Makes perfect sense, of course!

Calling a service without adding a reference in BizTalk 2006 and BizTalk 2006 R2

2008-11-29T23:17:00.001Z

We’ve been experimenting with calling ASMX web services from orchestrations without having to add a web reference (for the SOAP adapter) or use the generated items (for the R2 WCF adapter).

The idea, in short, is to achieve increased decoupling between systems even in a web service scenario -

Generally when you add a reference to a service in BizTalk 2006 or in R2 (although there are some clear differences between the implementation) the schemas for the request and response types are generated for you as well as an orchestration which defines message and port types using those schemas.

When using the SOAP adapter the types generated are somewhat “special” and they encapsulate a little bit of black magic; luckily the WCF adapter which shipped with R2 is much better in the sense that there’s nothing special about any of these artifacts (which also explains why it is now “Add Generated Items” and not “Add Service Reference” – as this is all it’s doing).

What this means is that if you follow the path that BizTalk leads you through you will get all these artifacts in the same assembly with your orchestration, which means you are now tightly coupled with the web service contract; not the end of the world, but if you want to stay true to the idea behind BizTalk - in which your processes can be masked from changes in the other applications you have to play pretend a little bit.

We thought that if we had the web service schemas in a separate assembly, and our process only used it’s own representation of the data (which would, ideally, be less than the entire data provided by the, mostly generic, web service) we could then map between the two in the port rather than in the orchestration, which would mean that if the web service changes, all we will need to do (in theory, at least) is re-deploy the assembly with service’s schemas assembly the and the map.

So – how I went about doing that with the WCF adapter -

Following best practice I had an assembly to hold all of “My” schemas – these are the ones describing entities in my domain.

I then created an orchestration assembly to contain my orchestration, which references the schemas assembly; the orchestration assembly has no other dependencies.

I then created a third assembly to include all the types for the service - I went through the “Add Generated Items” wizard to get all the artifacts, but I only really used the schemas (and not the message or port types); this assembly, like the schemas assembly, has no dependencies.

I then progressed to create a fourth assembly to hold the mapping between my schemas and the service’s schemas; naturally this assembly references both projects, but, crucially, it is referenced by no-one.

So – at the end of this we get the following -

I then imported the send port bindings generated by the wizard to create the send port; I could have quite happily created it from scratch as there’s nothing special in that port - with the exception of one point, discussed next - so this was really just to save me some time, and added the two maps I’ve created to map the process output format to the service request and the service response to the process input format.

Goal achieved – the process knows nothing about the service – all is done externally to the process through port configuration.

But did it work? Almost - running this scenario I received a soap fault from the service complaining about a misunderstood soap action; makes sense I thought – how would BizTalk know which service operation I wanted?

Well, the WCF adapter has a very nice way to figure out the soap action to use (in my view) – as part of the port configuration there’s a bit of xml that provides mapping between an orchestration send port’s operation name and the required soap action; the setting looks something like this -

In the generated port type the operation name matches the operation name in the service description (“HelloWorld”, in my example), which, in turn, is mapped through this xml to the relevant soap action; as I did not use the generated types the operation name did not match – I simply left it as the default “Operation_1” (naturally…); that meant that when the request came the adapter failed to find a matching operation.

Somewhat annoyingly, what the adapter does when it can’t resolve the name is to assume that the entire setting should be used as the soap action and so the entire xml was written to the header; this behaviour is there to allow one to specify a fixed header to use, but I think the experience could be a bit better there – they could have had two different settings, or at least realise that if I’ve put a BtsActionMapping xml in there I do not intend for it to be used as the header itself(!), and so, if the relevant entry was not found the request should be suspended rather than going out incorrectly to the service; never-the-less the operation could not be resolved, of course, and the service returned a soap fault.

Fixing the issue was easy and simply meant adding the correct entry in the xml and running the scenario again, this time it completed successfully.

How does that differ using the SOAP adapter?

Using the SOAP adapter the approach was naturally very similar; pretty much the same assemblies, pretty much the same artifacts; there are three key differences though -

For starters the soap adapter requires a proxy; in most scenarios you’re using a web port type which provides the adapter with a proxy and so in most cases you don’t have to worry about this at all; I can imagine that some are probably not even aware but the send port, using the SOAP adapter, will have the web service proxy set in the “Web Service” tab of the adapter configuration to “Orchestration Web Port”.

Alternatively you can provide a custom proxy class, which is a topic by itself (and you can check it out in Richard Seroter’s post on the topic here), but in most standard cases this is not required.

As I’m not following the “standard” approach I had to create a custom proxy for my send port; I did this by using WSDL.exe and configuring the proxy class in the send port as described in Richard’s post.

In my case, however, unlike Richard’s, I did not wish to pre-defined the method called in the send port; luckily – the configuration allows you to set it to “Specify Later”, which means the method name will be provided per request through the message context (using the SOAP.MethodName property).

Taking the “Specify Later” approach means I don’t have to have a send port per method, which is good of course, but pay attention to my note regarding number of ports in the summary below.

Now that I have the send port and proxy configuration sorted I needed to get the web service’s schemas; I could do that by using XSD.exe and add the output to my service types assembly.

Last thing – when using the soap adapter you don’t generally need to have an XmlDisassembler in the pipeline; however – if you want BizTalk to be able to run a map it needs a “proper” message type in the context, not that awkward one the SOAP adapter puts, and so the XmlDisassembler becomes mandatory in this scenario.

other than that everything else is pretty much the same.

So – to summarise –

Calling a service from a process, without the process knowing ANYTHING about the service implementation is very easy, the story is slightly better in the WCF adapter case in my view, but both seem to me quite reasonable.

The only downside to this approach that I could think of so far, is that you are likely to end up with as many send ports as you have response output formats -

As far as requests from the orchestrations to the web service are concerned BizTalk will quite happily pick up the right map from a list of configured maps based on the input; so - if process A has one output format and Process B has a different output format and they both share the same send port – BizTalk will pick up the relevant map to convert either outputs to the service’s request.

On the way back, however, the incoming message (the service’s response) always looks the same, and so BizTalk will have no way of knowing which map to pick from the list.

That means that multiple send ports will have to be created for such cases so that there’s only one map for the service’s response; a large number of those may have some impact on the overall performance of the server group as the number of subscriptions that need to be evaluated increases; what “large” means in this context and how big is the impact is not something I could say easily, so I’d suggest doing some benchmarking to find out in your environment if you are concerned.

Configuring the Geneva Framework based STS to work with custom UserNamePasswordValidator

2008-11-25T10:16:00.001Z

It took me a little while (and quite a bit of help from others on this thread) to get to a relatively simple implementation, so I thought I’d summarise the steps I’ve taken –

At the risk of sounding the obvious I would definitely recommend making sure the overall STS scenario works well using windows authentication before changing it to support custom authentication.

Once that’s done change the STS’ bindings’ clientCredentialType to UserName and the establishSecurityContext to false.

</security>

</binding>

</ws2007HttpBinding>

The equivalent changes need to be made on the clinet’s binding going to the STS. these may not be obvious at first glance – on the client you will have the endpoint representing the RP and using ws2007FederationHttpBinding; inside this binding’s configuration you will find the issuer element, which is somewhat similar to an endpoint; this represents the STS’ endpoint and as such has a binding (ws2007HttpBinding) and a binding configuration; it is in that binding’s configuration that you need to change the credential type. setting it on the wrong bindings, as I did initially would send you back a couple of hours :-)

Next, as we’re using username authentication for the STS, a service certificate must be used so that the credentials can be encrypted. This is done through configuration of a service behaviour on the STS service as such:

</serviceCredentials>

</behavior>

</serviceBehaviors>

(don’t forget to wire the behaviour to the service...)

As the test certificate I’m using are not valid, I needed to disable validation on the client side; I could not find a way to do this through configuration, as at the client there isn’t an endpoint as such for the STS service, just the issuer element in the ws2007FederationHttpBinding, so I’ve done this in the client code (this is a temporary measure for development only!) –

proxy.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None;

proxy.ClientCredentials.ServiceCertificate.Authentication.RevocationMode = X509RevocationMode.NoCheck;

The current version of the Geneva Framework, unlike Zermatt before it, does not support the userNameAuthentication element in the serviceCredetial service behaviour. (to be accurate you can kind of force it to do so, but that’s planned to be blocked in the near future, so for all intents and purposes you should not include this element, see more information in the thread mentioned above)

In order to implement authentication a customer SecurityTokenHandler needs to be added; to do so created a class that inherits from WindowsUserNameSecurityTokenHandler and overridden the ValidateToken method; again – several samples of such implementation exist on the thread but the idea is to validate the username and password (made available through the SecurityToken parameter to the method) in whatever way you wish and then, ideally, add some claims to the ClaimsIdentityCollection output; this should generally include the identity, authentication method and authentication instant, but you can add whatever you wish.

To wire the custom handler to the STS service a bit more configuration is required on the STS side -

<microsoft.identityModel>

</securityTokenHandlers>

</microsoft.identityModel>

This replaces the built-in WindowsUserNameSecurityTokenHandelr with my class that inherits from WindowsUserNameSecurityTokenHandler and adds custom implementation.

Note: I needed to add the definition of this section as such –

I hope that makes sense….

From "Zermatt" to the "Geneva Framework" part II

2008-11-06T09:58:00.001Z

A couple of days ago I've posted about the changes I've had to make to allow my custom STS to work with the updated Geneva framework. there's one more, quite crucial, change that I had to make, which I will try to describe next -

If my understanding is correct (and unfortunately there's all the chances in the world that it is not, so if you know otherwise please do comment) the October Geneva SDK has tightened security a little bit around token validation.

I believe that the previous version of SDK, the RP simply made sure that a token was included with the request, and that this token was signed by a party whose certificate exists on the server (and is accessible); the RP did not check which certificate was used to sign the token.

As far as I can tell the Geneva Framework SDK now behaves differently - if you execute the same code and configuration you had before (baring necessary changes to allow the code to compile on the new version, but these are mostly name changes) you will get the following error from the RP:

"An unsecured or incorrectly secured fault was received from the other party. See
the inner FaultException for the fault code and detail."

Basically the Client gets a token from the STS and attaches it to the request but the RP does not recognise the issuer of the token; in order to instruct the RP to accept tokens signed by a particular STS you need to provide it with a list of issuers you accetps, this can be done using the following configuration for example -

<microsoft.identityModel>
<issuerNameRegistry type="Microsoft.IdentityModel.Tokens.ConfigurationBasedIssuerNameRegistry, Microsoft.IdentityModel, Version=0.5.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
    <trustedIssuers>
      <add name="STS" thumbprint="7a0671d475673c1ab131ca1c0c804e4fbd385140"/>
    </trustedIssuers>
</issuerNameRegistry>
</microsoft.identityModel>

This bit of configuration lists all the certificates that are acceptable as STS token signing.

It is interesting to note that this model is completely extensible - you can define your own registry IssuerNameRegistry type that would look and behave differently if you have other means of listing those; the same can also be done via code, which is the example provided with the SDK - you define a custom IssuerNameRegistryClass -

namespace ClaimsAwareWebService
{
    public class TrustedIssuerNameRegistry : IssuerNameRegistry
    {
        /// <summary>
        /// Returns the issuer Name from the security token.
        /// </summary>
        /// <param name="securityToken">The security token that contains the STS's certificates.</param>
        /// <returns>The name of the issuer who signed the security token.</returns>
        public override string GetIssuerName( SecurityToken securityToken )
        {
            X509SecurityToken x509Token = securityToken as X509SecurityToken;
            if ( x509Token != null )
            {
                //Note: This piece of code is for illustrative purposes only. Validating certificates based on
                //subject name is not a good practice. This code should not be used as is in production.
                if ( String.Equals( x509Token.Certificate.SubjectName.Name, "CN=STS" ) )
                {
                    return x509Token.Certificate.SubjectName.Name;
                }
            }

            throw new SecurityTokenException( "Untrusted issuer." );
        }
    }
}

and then when configuring the host for the RP service you provide this as a parameter -

FederatedServiceCredentials.ConfigureServiceHost(host, new TrustedIssuerNameRegistry());

And while I'm on the subject - as this has sent me going in circles - it appears that the framework is not happy with claim-less tokens, so if you're dumb enough (as I was) and end up not adding any claims (I was adding them base on the requested claims in the incoming request, which, at some point, was empty in my configuration) you will get a error, which, after setting the ServiceDebugBehavior would read "A SamlAssertion requires at least one statement. Ensure that you have added at least one SamlStatement to the SamlAssertion you are creating."

I can't decide about this one - does it not make sense to have a scenario in which you just want to get a signed token to indicate that an STS has authenticated the caller, but don't actually need any claims? not that it's a problem to find at least one claim to add (identity, authentication method are two easy examples), but speaking in principal I'm not yet convinced not having any specific claim should be an error.