Bits of the old in-out in-out

Inserting an Entity into Azure Table Storage from BizTalk

noreply@blogger.com (Paul Arundel) — Fri, 15 Jan 2010 10:49:00 +0000

Following on from my previous post where I used a custom WCF adapter to insert a BizTalk message into Azure blob storage I wanted to try the same but with Azure Table storage. This would of course have been more meaningful if it had followed immediately after my previous post and not three months after. I blame Christmas.

An Azure table is not like the familiar relational tables we get with SQL Server. If you want one of those in the cloud try SQL Azure. Azure tables are schema-less and designed for scalability. As an individual table has no schema you can insert different entity types with different properties into the same table and it won't mind a bit. This allows you to store types that are naturally retrieved at the same time (say an Order and it's Order items) in the same table and optimize your queries appropriately. This is partly achieved by each table entity having two keys, a partition key and a row key, which jointly make up an entity's unique key. The partition key plays the important role in that entities with the same partition key will be stored consecutively on the same storage node, therefore avoiding queries having to scan multiple storage nodes for entities with a particular partition key. This is, at least, how I understand it. (This is also possibly irrelevant unless you are dealing with a lot of entity rows).

Inserting a record into an Azure table is detailed in the API here. This is similar to that detailed for inserting to blob storage: we set a few HTTP headers, include our data in the body of the message and make a REST like call. It differs in that the HTTP method is a POST rather than a PUT, additional HTTP headers are required and the message body (our entity data) has to be inserted as an Atom feed entry. So my first task was to map my message instance to the required Atom entry format:

As there are no defined XSD for the Atom format I had to create my own schema for the various elements/namespaces required. I'm a little frustrated with the schema I ended up with as it's quite unsophisticated and doesn't leave much room for re-use (i.e. wrapping entities other than my sample 'customer' schema):


<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://www.w3.org/2005/Atom" elementFormDefault="qualified" targetNamespace="http://www.w3.org/2005/Atom" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:import schemaLocation=".\dataservicescustomermeta.xsd" namespace="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" />
  <xs:annotation>
    <xs:appinfo>
      <b:references>
        <b:reference targetNamespace="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" />
        <b:reference targetNamespace="http://schemas.microsoft.com/ado/2007/08/dataservices" />
      </b:references>
    </xs:appinfo>
  </xs:annotation>
  <xs:complexType name="AtomEntryType">
    <xs:sequence>
      <xs:element name="title" type="xs:string" />
      <xs:element name="updated" type="xs:string" />
      <xs:element name="author">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="name" type="xs:string" />
          </xs:sequence>
        </xs:complexType>
      </xs:element>
      <xs:element name="id" type="xs:string" />
      <xs:element name="content">
        <xs:complexType>
          <xs:sequence>
            <xs:element ref="m:properties" />
          </xs:sequence>
          <xs:attribute default="application/xml" name="type" type="xs:string" />
        </xs:complexType>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:element name="entry" type="AtomEntryType" />
</xs:schema>

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices" elementFormDefault="qualified" targetNamespace="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="RowKey" type="xs:string" />
  <xs:element name="PartitionKey" type="xs:string" />
  <xs:element name="TimeStamp" type="xs:string" />
  <xs:element name="FirstName" type="xs:string" />
  <xs:element name="LastName" type="xs:string" />
  <xs:element name="Email" type="xs:string" />
</xs:schema>

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" elementFormDefault="qualified" targetNamespace="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:import schemaLocation=".\dataservicescustomer.xsd" namespace="http://schemas.microsoft.com/ado/2007/08/dataservices" />
  <xs:annotation>
    <xs:appinfo>
      <b:references>
        <b:reference targetNamespace="http://schemas.microsoft.com/ado/2007/08/dataservices" />
      </b:references>
    </xs:appinfo>
  </xs:annotation>
  <xs:element name="properties" type="PropertiesType" />
  <xs:complexType name="PropertiesType">
    <xs:sequence>
      <xs:element ref="d:PartitionKey" />
      <xs:element ref="d:RowKey" />
      <xs:element ref="d:TimeStamp" />
      <xs:element ref="d:FirstName" />
      <xs:element ref="d:LastName" />
      <xs:element ref="d:Email" />
    </xs:sequence>
  </xs:complexType>
</xs:schema>

To configure the send port we can directly set the port URL to the table end point address http://[accountname].table.core.windows.net/[tablename]. Unlike Blob storage we don't need to include a unique Id in the URL we just need to use the table address. Therefore my custom WCF behavior does not need to use manual addressing in this case nor does it need to set the HTTP method as it is POST by default.

This simplifies the code in the behavior somewhat (Apart from the two methods below, the code remains as my previous post).


public object BeforeSendRequest(ref Message request, IClientChannel channel)
{
    HttpRequestMessageProperty httpProp = GetOrCreateHttpProperty(request);
    
    string contentType = "application/atom+xml";

    //we need the same date in both the auth string and http headers
    DateTime date = DateTime.Now;

    string authorizationHeader =
        AzureAuthorizationHeader.GetAuthorizationString(_azureAccountName,
                         Convert.FromBase64String(_azureAccountKey),
                         _httpMethod,
                         contentType,
                         date,
                         channel.RemoteAddress.Uri.AbsolutePath);
                                                                
          
    httpProp.Headers.Add("Authorization", authorizationHeader);
    httpProp.Headers.Add("x-ms-date",AzureAuthorizationHeader.GetFormattedDate(date));
    httpProp.Headers.Add("Content-Type", contentType);
    httpProp.Headers.Add("DataServiceVersion", "1.0;NetFx");
    httpProp.Headers.Add("MaxDataServiceVersion", "1.0;NetFx");
    httpProp.Headers.Add("Accept", "application/atom+xml,application/xml");
    httpProp.Headers.Add("Accept-Charset", "UTF-8");
    return null;
}


public static string GetSignatureString(string accountName,
                                        string method,
                                        string contentType,
                                        DateTime date,
                                        string resourcePath)
{
    string CanonicalizedResource = string.Format("/{0}{1}",accountName,resourcePath);

    return string.Format("{0}\n{1}\n{2}\n{3}\n{4}",
                         method.ToUpper(),
                         string.Empty, //ContentMD5 empty
                         contentType,
                         GetFormattedDate(date), //can use x-ms-Date
                         CanonicalizedResource);
}

A few other changes from Blob storage. We need to set additional headers as shown in the code. The authorization header string is also structured differently and does not include the CanonicalizedHeaders part.

Finally let's set Fiddler as a proxy on the send port and see what we send on the wire on a successful insert:

Using a BizTalk WCF-Custom adapter to directly send messages to Azure Storage

noreply@blogger.com (Paul Arundel) — Wed, 28 Oct 2009 22:12:00 +0000

In this post I want to look at sending a message from BizTalk directly to Azure blob storage. Accessing Azure Storage is exclusively via a REST API so we should hopefully be able to do this with the standard adapters (HTTP or WCF) and not need to resort to writing our own custom adapters.

To store a message in Blob storage, the API defines the destination URL as: http://myaccount.blob.core.windows.net/mycontainer/myblob. Presuming we already have the a blob storage container created it doesn't seem unreasonable to set the URL including the container in the send port config. However the 'myblob' part will need to be our unique key for the blob so we will have to set that programmatically. We will also need to set the HTTP method to 'PUT' rather than POST (the default) to store our message. Finally the API defines a number of required HTTP headers: Authorization, Date (or x-ms-date), Content-Length and Content-Type.

Using the HTTP adapter to get around these issues would appear to be awkward. We could set the HTTP headers by using the HTTP Adapter Property 'UserHttpHeaders' in an orchestration or custom pipeline. However the Authorization header encoded string needs to include (see later) our destination URL, something I want to have stored in our port. We could try a dynamic port (and annoyingly store our URL config elsewhere) but we would then have to get around the problem of the HTTP adapter using the HTTP 'POST' method.

WCF's configuration and extensibility options appear to be a better option. Though the obvious choice, using the 'REST friendly' webHttpBinding, does not appear to work. I could not get a simple BizTalk WCF-Custom send port, using the webHttpBinding (with webHttpBehavior added) and all default config to send a message of any kind without error. A few blog/forum posts relating to 2006 R2 indicate the binding is not supported (due to the contracts BizTalk defines). Presumably it's still not supported for 2009 but maybe I'm missing something?

Happily I found Jon Flanders presentation on using BizTalk 2006 R2 with a REST service (the approach is also detailed here). Without going into the full details of the presentation, he wants to use a WCF send adapter to send a 'GET' request to a REST service. Firstly he uses a customBinding with HTTPTransport to get around BizTalk not playing nice with the webHTTPBinding and sets the MessageVersion to 'None' to remove the SOAP wrapper. Secondly he codes up a WCF behavior extension to set the HTTP method to 'GET' as the message is passed through the WCF channel stack. This is exactly what I need to do.

The code for my WCF behavior extension is further below, but I will show you the BizTalk send port configuration:

The next dialog shows the customBinding with TextMessageEncodingElement and HttpTransportElement (note manualAddressing is set to 'true'):

Configuring the WCF behavior extension I have added:

The code for the behavior performs a few tasks and is all completed in the BeforeSendRequest method of a IClientMessageInspector. First it sets the HTTP method to the supplied verb as in John's presentation. It then creates a new destination address by concatenating the current remote address with a unique ID acting as the blob storage key. I use a GUID for the key, but for BizTalk messaging this could be updated to something more useful, possibly from the message data. The new address is then set on the request.Headers.To property. This can change the destination address as having manual addressing set to 'true' allows us to change the address after the channel stack has been created. I then create the signature for the Authorization header (see below) and set the other HTTP headers required by Azure Storage. (note: I haven't shown the other empty interface methods)


public class AzureStorageBehavior : IEndpointBehavior, IClientMessageInspector
{
    public string _httpMethod;
    public string _azureAccountName;
    public string _azureAccountKey;

    public AzureStorageBehavior(string method, string account, string key)
    {
        _httpMethod = method;
        _azureAccountName = account;
        _azureAccountKey = key;
    }

    public object BeforeSendRequest(ref Message request, IClientChannel channel)
    {
        HttpRequestMessageProperty httpProp = GetOrCreateHttpProperty(request);
        httpProp.Method = _httpMethod;

        //todo just create a guid blob item name for the moment
        string id = Guid.NewGuid().ToString();

        //manually address
        request.Headers.To = new Uri(channel.RemoteAddress.Uri, id);

        //todo set content
        string contentType = "text/xml";

        //we need the same date in both the auth string and http headers
        DateTime date = DateTime.Now;

        string authorizationHeader =
            AzureAuthorizationHeader.GetAuthorizationString(_azureAccountName,
                                                            Convert.FromBase64String(_azureAccountKey),
                                                            _httpMethod,
                                                            contentType,
                                                            date,
                                                            request.Headers.To.AbsolutePath);
        //required for Azure Blob Storage
        httpProp.Headers.Add("Authorization", authorizationHeader);
        httpProp.Headers.Add("x-ms-date", AzureAuthorizationHeader.GetFormattedDate(date));
        httpProp.Headers.Add("Content-Type", contentType);

        return null;
    }

    public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {
        clientRuntime.MessageInspectors.Add(this);
    }
    //snip
}
public class AzureStorageBehaviorExtension : BehaviorExtensionElement
{
    protected override object CreateBehavior()
    {
        return new AzureStorageBehavior(HTTPMethod, AzureAccountName, AzureAccountKey);
    }

    [ConfigurationProperty("httpMethod")]
    public string HTTPMethod
    {
        get { return (string)base["httpMethod"]; }
        set { base["httpMethod"] = value; }
    }

    [ConfigurationProperty("azureAccountName")]
    public string AzureAccountName
    {
        get { return (string)base["azureAccountName"]; }
        set { base["azureAccountName"] = value; }
    }

    [ConfigurationProperty("azureAccountKey")]
    public string AzureAccountKey
    {
        get { return (string)base["azureAccountKey"]; }
        set { base["azureAccountKey"] = value; }
    }

    public override Type BehaviorType
    {
        get { return typeof(AzureStorageBehavior); }
    }

}

The code below shows the few static methods I have created to form the Authorization signature for the HTTP header. The code is a minimal implementation to meet the requirements as detailed in the Authentication Schemes reference and from looking at the StorageClient sample in the Azure SDK. It's mostly a matter of concatenating the various required elements and signing the resultant string with the Azure key:


public class AzureAuthorizationHeader
{
    public static string GetAuthorizationString(string accountName,
                                        byte[] key,
                                        string method,
                                        string contentType,
                                        DateTime date,
                                        string resourcePath)
    {
        string signatureString = GetSignatureString(accountName, method, contentType, date, resourcePath);

        return string.Format("SharedKey {0}:{1}", accountName, EncodeSignature(signatureString, key));
    }

    //from api sample
    public static string EncodeSignature(string signatureString, byte[] key)
    {

        byte[] dataToMAC = System.Text.Encoding.UTF8.GetBytes(signatureString);

        using (HMACSHA256 hmacsha1 = new HMACSHA256(key))
        {
            return System.Convert.ToBase64String(hmacsha1.ComputeHash(dataToMAC));
        }
    }

    public static string GetSignatureString(string accountName,
                                            string method,
                                            string contentType,
                                            DateTime date,
                                            string resourcePath)
    {
        //just the date header 
        string canonicalizedHeaders = string.Format("x-ms-date:{0}", GetFormattedDate(date));
        string CanonicalizedResource = string.Format("/{0}{1}", accountName, resourcePath);

        return string.Format("{0}\n{1}\n{2}\n{3}\n{4}\n{5}",
                               method.ToUpper(),
                               string.Empty, //ContentMD5 empty
                               contentType,
                               String.Empty, //Date HTTP header empty - have to use x-ms-Date
                               canonicalizedHeaders,
                               CanonicalizedResource);
    }

    public static string GetFormattedDate(DateTime date)
    {
        return date.ToString("R", CultureInfo.InvariantCulture);
    } 
}

As I've used a standard WCF behavior to achieve this, it doesn't have to be used along with BizTalk and could be used just as easily from a normal WCF client. But in either case the entire mechanism is abstracted away within the WCF endpoint configuration.

Using an Azure Worker Process to Read and Store Messages from a Service Bus Queue

noreply@blogger.com (Paul Arundel) — Wed, 30 Sep 2009 14:42:00 +0000

Well, the forthcoming October updates to the .NET Services CTP have just about scuppered the project I have been working on and blogging about. Routers are being 'Temporarily' removed, Queues are being replaced with 'Message Buffers' and we can no longer subscribe external endpoints to Routers, doing away with the means to push messages to services outside of the service bus. Presumably these features have been temporarily dropped so the .NET Services developers can focus on the remaining features for the first release. But in that case why replace Queues with the new Message Buffers? And when Routers, which share a common API to Queues, re-appear will they also be in a changed form? No update yet from Clemens.

Anyway I still want to continue getting a feel for running processes in the Azure cloud and the possibilities it offers. For this post I'm going to switch my attention to the Windows Azure part of the Windows Azure Platform and take a look at worker roles and storage. In my last post I looked at an external client retrieving messages from a .NET Services Queue which was, in turn, receiving messages from a Router. Now my aim is to have a worker process running on the Azure cloud pulling messages from a .NET Services Queue at regular intervals and securely storing the messages with the Azure Blob service. This could act as a message archiving service for backup or auditing requirements and could be easily hooked up to any process using .NET Services Routers. It is similar in aim to Mike Stephenson's Azure Pipeline Component for BizTalk, though in this case all the processes are running in the cloud: message routers, queues and storage. Taking the diagram from my last post I will now add an additional process to archive any messages published to the router:

I will use an Azure worker role to continuously check the Queue for messages. A worker role is basically a background process running in the cloud that does not expose any endpoints and therefore executes at its own schedule. Once the Azure tools are installed, you can create a worker process project by using one of the cloud service templates, 'Worker Cloud Service' in this case. This creates some sample code for your worker process:


public class WorkerRole : RoleEntryPoint
{
    public override void Start()
    {
        // This is a sample worker implementation. Replace with your logic.
        RoleManager.WriteToLog("Information", "Worker Process entry point called");

        while (true)
        {
            Thread.Sleep(10000);
            RoleManager.WriteToLog("Information", "Working");
        }
    }

    public override RoleStatus GetHealthStatus()
    {
        // This is a sample worker implementation. Replace with your logic.
        return RoleStatus.Healthy;
    }
}

My worker process, will need to complete two tasks, check the Service Bus queue for new messages and store any new messages in blob storage. I've implemented a couple of simple classes to do these tasks and updated the worker process code:


public override void Start()
{
    RoleManager.WriteToLog("Information", "Worker Process entry point called");
    QueueReader qr = new QueueReader("bitsinout", "/ProductUpdate/QueueSubscriber/");
    MessageArchive ma = new MessageArchive("testarchive");
    while (true)
    {
        Thread.Sleep(10000);
        RoleManager.WriteToLog("Information", "Checking Queue.");
        string response = qr.CheckQueue();
               
        if (!String.IsNullOrEmpty(response))
        {
            RoleManager.WriteToLog("Information", response);
            ma.StoreMessage(Guid.NewGuid().ToString(),response);
        }
    }
}

My QueueReader class uses the .NET Services API and helper functions as in my last post. I construct the QueueReader with the name of my .NET Services solution and the path to the Queue we are interested in. Obviously this information should really be retrieved from a configuration store. It also wouldn't be too much work to supply a number of queue addresses enabling a single worker process to check multiple queues. (Note: when using the .NET Services API from a worker role project you will get a 'partially trusted' assembly error unless you set enableNativeCodeExecution="true" in the ServiceDefinition.csdef file.)


public string CheckQueue()
{
    QueueClient client = GetOrCreateQueue(_queueUri);
    string response = null;

    try
    {
        Message m = client.Retrieve(new TimeSpan(0, 0, 5));
        response = m.GetBody<string>();
    }
    catch (TimeoutException){}

    return response;
}

For simplicity I use the client.Retrieve() method but as discussed before we could retrieve multiple messages or use the Peek/Lock methods for better reliability. I'll have to update this to use Message Buffers when the new CTP arrives.

Finally my MessageArchive class stores the message (a string) in blob storage. I use the StorageClient sample provided with the Azure SDK to store my messages. This sample is actually a wrapper around a REST interface which is currently the only way to interact with azure storage. The StorageClient is strictly not an API at the moment, but probably gives us a good indication of what an official .NET API will look like.


public class MessageArchive
{
    BlobContainer _container;
    public MessageArchive(string containerName)
    {
        StorageAccountInfo account = 
          new StorageAccountInfo(new Uri("http://blob.core.windows.net/"),
                                 null, 
                                 "bitsinout",
                                 "*************************************8");

        BlobStorage blobStorage = BlobStorage.Create(account);
        _container = blobStorage.GetBlobContainer(containerName);
        //Create the container if it does not exist.
        if (!_container.DoesContainerExist())
        {
            _container.CreateContainer(null, ContainerAccessControl.Private);
        }
           
    }

    public void StoreMessage(string name, string message)
    {
        _container.CreateBlob(new BlobProperties(name), new BlobContents(Encoding.UTF8.GetBytes(message)), true);
    }
}

Note the URI passed to the StorageAccount does not include the sub domain name of your Azure storage solution.

Let's run this process in the dev fabric, using my .NET Services publisher runtime to populate the Queue. The Development Fabric UI shows the worker role logging the periodic checking of the queue for messages:

As I had created a web role along with the worker role I created a quick aspx page to list the contents of the Blob container (first ASP.NET I've done in a few years!):

The final step is to deploy and run the solution to the Azure cloud. As others have noted this is a simple process. Select 'Publish' on the cloud service project in Visual Studio. This will create a package file ([projectname].cspkg) and a config file (ServiceConfiguration.cscfg). Browse to your service on the Windows Azure portal, click 'deploy', select the two files above and once uploaded you can run your project:

I used my runtime to publish a few messages to the queue and checked my web page which is now hosted in Azure. I can see the new messages (with a later timestamp) have been stored, proving my worker process is now running happily in the clouds:

.NET Services - Queues

noreply@blogger.com (Paul Arundel) — Sun, 13 Sep 2009 10:41:00 +0000

Along with pub/sub routing functionality, the Azure Service Bus provides an additional messaging primitive: Queues. Where a Router will immediately push messages to subscribers, a Queue will hold messages in persistent storage until retrieved by a consumer (as we would expect!). This disconnects our publisher and consumer and allows the consumers to retrieve messages reliably at their own schedule. Queues can be added as subscribers to a Router much like any other endpoint, allowing us to compose the two messaging primitives within the Service Bus.

Continuing the theme from my previous posts, I now wish to add a Queue as a subscriber to my published message. This handles the scenario where the consumer does not wish to immediately receive messages as they are published. I now have three subscribers:

For the first two subscribers (a WCF endpoint and a simple http endpoint) the Router will immediately forward on any published messages. For our Queue subscriber, the published message will be stored on a Queue and the subscriber/consumer will retrieve the message at their own schedule.

So how do we add a Queue subscription to a Router? As before the API makes this relatively simple:


string qServicePath =  string.Format("/{0}/{1}/",subscriber.Publisher.Name,subscriber.Name);
Uri qSbURI = ServiceBusEnvironment.CreateServiceUri("sb", subscriber.Publisher.Solution.Name,qServicePath);
QueueClient qClient = ServiceBusHelper.GetOrCreateQueue(qSbURI);
qClient.SubscribeToRouter(_routerClient,TimeSpan.FromHours(1));

First we create an appropriate path for our Queue, in this case sb://bitsinout.servicebus.windows.net/ProductUpdate/QueueSubscriber. We then create a QueueClient by either creating the queue or retrieving a reference to any existing queue at that URL. Finally we use the Queueclient SubscribeToRouter() method, passing in a reference to our existing RouterClient instance. The GetOrCreateQueue is static helper method and is very similar to the method for creating a Router:


public static QueueClient GetOrCreateQueue(Uri queueUri)
{
    TransportClientEndpointBehavior serviceBusCredential = GetCredential();
    QueuePolicy policy;
    QueueClient client;
    try
    {
        client = QueueManagementClient.GetQueue(serviceBusCredential, queueUri);
        policy = client.GetPolicy();
    }
    catch (EndpointNotFoundException)
    {
        policy = new QueuePolicy();
        policy.Discoverability = DiscoverabilityPolicy.Managers;
        policy.Authorization = AuthorizationPolicy.NotRequired;
        policy.TransportProtection = TransportProtectionPolicy.None;
        policy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);
        client = QueueManagementClient.CreateQueue(serviceBusCredential, queueUri, policy);
     }

     return client;
}

Alternatively we can create a subscription to the Queue as we have with the other subscriptions rather than using the QueueClient.SubscribeToRouter() method. To do this we need to supply the address of the Queue with the Https protocol when adding the subscription:


string qServicePath =  string.Format("/{0}/{1}/",subscriber.Publisher.Name,subscriber.Name);
Uri qSbURI = ServiceBusEnvironment.CreateServiceUri("sb", subscriber.Publisher.Solution.Name,qServicePath);
Uri qHttpsURI = ServiceBusEnvironment.CreateServiceUri("https", subscriber.Publisher.Solution.Name, qServicePath);
QueueClient qClient = ServiceBusHelper.GetOrCreateQueue(qSbURI);
RouterSubscriptionClient subscriptionClient = _routerClient.Subscribe(new EndpointAddress(qHttpsUR), TimeSpan.FromHours(1));

The code for our consumer client is just as simple:


Uri queueUri = ServiceBusEnvironment.CreateServiceUri("sb", "bitsinout", "/ProductUpdate/QueuePush/");
QueueClient client = ServiceBusHelper.GetOrCreateQueue(queueUri);
//throws TimeoutException if no message in queue
Message m = client.Retrieve(new TimeSpan(0, 0, 5));
Console.WriteLine(m.GetBody<string>())

We get the Queue reference as before and call the Retrieve() method which makes a destructive read on the queue (the message is deleted from the queue when it is read). The method returns a WCF Message instance from which we can retrieve the message content. As an alternative to the Retrieve() method we can use RetrieveMultiple() to get all the messages from the queue:


var messages = client.RetrieveMultiple(int.MaxValue,new TimeSpan(0,0,10));
foreach(Message m in messages)
{
    Console.WriteLine(m.GetBody<string>());
}

Using the runtime and Quadrant as detailed in my last post I've set up the router subscriptions as shown in the diagram above. I've added a simple field to the model to allow a subscriber to be defined as a queue. The runtime creates a unique URL from the publisher and subscriber names:

Executing my runtime with these configuration values defined and the various services running I get the expected results as shown in the confusing screen grab below. The top console window shows the runtime process, creating the router and adding subscriptions. The next console window shows the WCF service receiving the messages and the following folder shows the messages written to file by the simple .aspx service. Finally the last console window shows the client process that retrieves the messages from the Queue at a slightly later time.

The Retrieve method is potentially risky as any failure in our client after the destructive read could result in the loss of the message. However we can add a level of reliability by using the PeekLock method as an alternative to Retrieve. Clemens Vasters discusses this in the context of an Audit Service where the reliable storing of the message is more important than the speed of transfer. PeekLock allows us to retrieve and lock the message in the queue. If we successfully complete our operation, we delete the message from the queue, otherwise we release the message in the queue allowing it to be retrieved again. The following code shows this process:


Message m = client.PeekLock(new TimeSpan(0,0,20));
string mb =  m.GetBody<string>();
Console.WriteLine(string.Format("{0} - Message retrieved - {1}", DateTime.Now,mb));
try
{
    File.WriteAllText(args[0], mb);
    client.DeleteLockedMessage(m);
    Console.WriteLine(string.Format("{0} - Message deleted", DateTime.Now));
}
catch (Exception e)
{
    Console.WriteLine(string.Format("{0} - Error - {1}", DateTime.Now,e.Message));
    client.ReleaseLock(m);
    Console.WriteLine(string.Format("{0} - Message released", DateTime.Now));
}

The following screen grab shows this process executed two times, the first with an invalid file path causing an error and the message to be released. If I hadn't called the ReleaseLock method Azure Service Bus takes care of this situation by automatically releasing locked messages after one minute.

Mixing it up Part II: Editing data with Quadrant and runtime retrieval of data from the Oslo Repository

noreply@blogger.com (Paul Arundel) — Wed, 26 Aug 2009 10:31:00 +0000

In my last post I hand coded M values as an instance of my model for configuring .NET Services router subscriptions. As I had not yet uploaded the model or the instance data to the repository that would be the next step. Without using the Oslo command line tools the easiest way to do this is to open your model definition m file in Intellipad and select M Mode -> T-SQL Preview. Presuming your model is error free this opens a new panel with various SQL statements defining your model in terms of SQL CREATE TABLE statements and the relationships between them. If you include instance data for your model, appropriate INSERT statements will also be listed. This text can then simply be pasted into a Management Studio query and executed.

Now we can fire up Quadrant and have a look at our newly defined tables and data:

Let's try dragging the selected 'Subscribers' item to a separate window. This displays a list of the data in the Subscribers table which we can edit:

Select the 'insert item' menu option to add a new subscriber to the table:

Oh. Quadrant dynamically uses the model relationships to generate the views/editing actions that we can take. In this case, to add a new Subscription we need to know the related Publisher and Quadrant cannot, of course, determine this from a list of the subscriptions table's contents. To get the appropriate context we will need to drill down from the Publisher to create the subscriptions. To make things even clearer, I dragged the Solutions data from the intial folder window and selected the 'Tree Master/Detail' view option:

Now the 'Insert Item' option works happily and I can define a new Subscription. Additionally I can't help but notice that the 'Tree Master/Detail' view when selected at the top level of a solution looks very similar to the DSL I had defined initially with MGrammar.

My runtime uses the Entity Framework to get the repository data. A quick tidy up of pluralization issues and we have an object model to work with:

With this created we can now define a simple method to retrieve the needed subscriptions when a process wishes to publish a message (this presumes publisher names are unique):


private Publisher GetPublisher(string publisherId)
{
    using (RepositoryEntities context = new RepositoryEntities())
    {
        try
        {
            return context.Publishers.AsPublisherAggregate().Where(x => x.Name == publisherId).First();
        }
        catch (InvalidOperationException)
        {
            //handle error
            throw new Exception(string.Format("Publisher '{0}' not defined.",publisherId));
        }
    }
}

Now we have the publisher data we can create the router and subscriptions with similar code to used in my previous post and the SDK samples. This immediately raises a number of question on how you would manage routers and subscriptions when publishing to the service bus. Create and delete them dynamically each time you publish a message? Create them once with a max expiration and periodically update the subscriptions to match data in the repository? For the moment my runtime determines if a router already exists, and if so, retrieves a list of current subscribers and adds any new subscribers to the router. The following method retrieves a list of current subscriber's URIs for a router


private List<Uri> GetCurrentSubscriptions(Uri routerUri)
{
    List<Uri> currentSubscriptions = new List<Uri>();
            
    HttpWebRequest req = HttpWebRequest.Create(new Uri(routerUri,"!(router/subscriptions)")) as HttpWebRequest;
    req.Method = "GET";
    req.Headers.Add("X-MS-Identity-Token", HttpGetAuthenticationToken());

    using (var r = req.GetResponse() as HttpWebResponse)
    {
        using (var rs = r.GetResponseStream())
        {
            Atom10FeedFormatter ff = new Atom10FeedFormatter();
            ff.ReadFrom(new XmlTextReader(rs));
            var links = ff.Feed.Items.SelectMany(x => x.Links).Where(l => l.RelationshipType == "alternate");
            links.ToList().ForEach(l=>currentSubscriptions.Add(l.Uri));
        }
    }

    return currentSubscriptions;
}

Finally the code for publishing a message. I use fixed values for the messaging format, obviously this could be configurable within the repository on a per publisher basis:


MessageVersion messageVersion = MessageVersion.Soap12WSAddressing10;
Binding binding = new WSHttpBinding(SecurityMode.None);
EndpointAddress epAddress = new EndpointAddress(_routerUriHttp);
Message m = Message.CreateMessage(messageVersion, "", message);

ChannelFactory<IOutputChannel> factory =  new ChannelFactory<IOutputChannel>(binding, epAddress)
factory.Endpoint.Behaviors.Add(GetCredential());
IOutputChannel channel = factory.CreateChannel();
channel.Open();
channel.Send(m);

With this defined lets try publishing a simple string of XML using the runtime. I have two subscribers defined in the Repository and matching services listening on my machine (a WCF host, and an .aspx page that saves any posted data to a file):

My next step is to look at handling a message transform for a particular subscriber and therefore how we can chain routers and subscriptions to provide more of a process flow. I never looked at the workflow capability that was originally part of .NET Services but that may well have been a candidate. As the workflow capability was removed an Azure WCF service that calls back into the service bus may provide a solution.

Mixing it up: Using the Oslo tools to define .NET Services Router Subscriptions

noreply@blogger.com (Paul Arundel) — Mon, 10 Aug 2009 18:46:00 +0000

My projects/blogging have recently focused on the Azure .NET Services platform and the various Oslo tools. So naturally I've arrived at the conclusion that I should attempt to combine these two strands of technology in my next project. When working with the .NET Services Service Bus I thought it may be helpful to define router subscriptions with a DSL using the Oslo tools. A process that needs to publish a message could then hand this responsibility to a runtime/service which would create the correct subscriptions for that publisher from the DSL defined data and send the message.

As my BizUnit DSL runtime parsed the object graph generated from my DSL, this time I wanted my runtime to retrieve the subscriptions data directly from the repository, the other major part of Oslo (much like Mikael Håkanssons with his WCF service configuration). To do this, as well as defining a DSL, I needed to create a model for my subscriptions with the M modelling language, use the Oslo tools to convert my DSL input file into an M instance of this model and import this into the repository. I could then toy with Quadrant as a way of editing my subscriptions. Should be easy......

Fairly quickly I had my grammar defined and a simple DSL input file, which is not much more than a list of subscribers:

The DSL simply defines the Service Bus solution name, publishers and the URIs of subscribers. Notice I have a Transform in there also. I intend my runtime to dynamically create subscriptions so that additional steps (such as a message transforming external service) can be performed. The Transform just serves as an example of this.

Next I defined my model using the MSchema modelling language, again super simple to start with:

Now my next step was to use the Oslo tools to parse my DSL input file and create M values of the input suitable for my model (following Dana Kaufmans posts). After a few errors using the command line tools I decided to hand code some working M values which would give me a better view of how my DSL grammar should project the input file. My hand coded M values could be defined a number of ways, including separate values for each of the collections (e.g. Transforms, Subscribers) related with label names. However the image below shows the values as one statement:

Obviously something was not right, how could I amend the projection of my DSL to look something like the above? Searching the forums I came across a number of posts from a few months ago by Rockford Lhotka discussing this issue (here and here and a short response from someone on the Oslo team). My model is defined using relationships between the entities, for example each Subscriber is related to one Publisher. In the generated SQL this becomes a foreign key from the Subscriber (child) to the Publisher table (parent). If we look at my DSL, the structure that is generated by mgx (and the Intelipad preview) is different. The relationship is defined from the parent to the children and is an hierarchical form like an XML document (e.g the relationship is defined from the Publisher (parent) to the subscribers (children)). No amount of amending the projection of my DSL was going to help.

As Rockford suggests in one of the posts above, one solution is to amend my Mschema model and use a hierarchical MSchema. In this way I would define the relationships from parent to child and hopefully then I could transform my input file into appropriate M values:

Unfortunately the SQL created from this model does not create the SQL as cleanly as the relational model and includes a join table. The Oslo SDK also notes "Do Not Use Collections to Model One-to-Many Relationships".

Discarding using hierarchical MSchema I have a number of choices to continue my project. I could write a new runtime to replace mgx for my dsl, traverse the object graph of a parsed DSL input file and create the appropriate M values. This seems a little pointless as I may as well just work directly from the object graph to create my router subscriptions. Alternatively I could stick with my original model and add my required configuration values with Quadrant. My subscriptions runtime could then run from the repository as I had intended. The latter is the sensible approach as this will let me move on to the fun part of my project and hopefully the DSL -> M -> Repository picture will become clearer in the future. However I feel slightly disappointed that I cannot demonstrate directly going from my DSL to the runtime via the repository, as the DSL provides such a simple and clear picture of the process defined.

This also raises the question of why you would use both a DSL and Quadrant. If you can execute your process from an instance of your DSL why use the repository? And if M/Repository/Quadrant provide a complete way to model and edit your data why use a DSL? As I'm learning a lot from Rockford Lhotka's various posts on this he raised this issue months ago starting an interesting discussion.

.NET Services - Routers - Pushing Messages to Internet Facing Services

noreply@blogger.com (Paul Arundel) — Tue, 28 Jul 2009 12:07:00 +0000

Continuing my look at .NET Services routers, I wanted the Service Bus to push a message to a public HTTP endpoint. Something that is not currently, as far I can see, in the SDK samples.

To avoid any reliance on the various web service technologies and to some extent to test platform independance I created a very simple endpoint using an .aspx page. This .aspx page simply writes out the message body to a file from a http POST. For what it's worth here's the code in Page_Load:


StreamReader sr = new StreamReader(Page.Request.InputStream);
File.AppendAllText(@"c:\dev\out\postdata.txt", sr.ReadToEnd());

I also re-configured my hardware router to forward port 80 requests to my server to ensure the service bus could talk to the service.

I then made a change to my publisher code to add a subscription for this endpoint, as unlike my ealier relay example, this endpoint will not make it's own. The following code was added after the code to create/get the RouterClient:


client.Subscribe(
    new EndpointAddress(@"http://myipaddress/testep/default.aspx"),
    TimeSpan.MaxValue);

Unfortunately this failed with an unhelpful 'internal error'. Experimenting I noticed that if I used a https in the subscription, the method call worked and my firewall log indicated an attempted connection on port 443 from the Service Bus, so the router and subscription were working. Failing to spot the obvious my progress was halted until I read this very helpful thread. By default the RouterPolicy.TransportProtection is set to TransportProtectionPolicy.AllPaths, indicating that the Router should use a secure connection when sending messages to subscribers. I therefore changed the policy to the only alternative:


policy.TransportProtection = TransportProtectionPolicy.None;

Executing the publisher process again and the service bus successfully called my simple service, writing the request to a file:


<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing">
    <s:Header>
        <a:Action s:mustUnderstand="1">http://bitsinout.servicebus.windows.net/IProductServices/ProductUpdate</a:Action>
        <a:To s:mustUnderstand="1">sb://bitsinout.servicebus.windows.net/productrouter/</a:To>
    </s:Header>
    <s:Body>
        <ProductUpdate xmlns="http://bitsinout.servicebus.windows.net">
            <product xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
                <Code>12345</Code>
                <Name>Oxtail Soup</Name>
                <Price>79</Price>
            </product>
        </ProductUpdate>
    </s:Body>
</s:Envelope>

So we have the SOAP message sent to the service, a good test that the router and subscription mechanism is working. The next step was to add a second subscription this time to a normal WCF service.


client.Subscribe(
    new EndpointAddress(@"http://myipaddress/ProductService"),
    TimeSpan.MaxValue);

As our RouterClient appears to give us a SOAP 1.2 message I used a wsHttpBinding for the service:


<service name="Arundel.NetServicesTest.ExternalWCFService.ProductService">
    <endpoint address="sb://bitsinout.servicebus.windows.net/productrouter/"
              listenUri="http://localhost/ProductService"
              binding="wsHttpBinding"
              bindingConfiguration="customWSBindingConfig"
              contract="Arundel.NetServicesTest.Services.IProductServices" />
</service>

<bindings>
    <wsHttpBinding>
        <binding name="customWSBindingConfig">
            <security mode="None">
                <transport clientCredentialType="None" />
            </security>
        </binding>
    </wsHttpBinding>
</bindings>

After initial failures, I created a custom binding configuration to remove security and authentication from the process, something to look at later. Once these complications were removed I could see from the WCF traces that I was getting an AddressFilterMismatch error, that the Action:To was the address of the Service Bus router rather than the intended service address. As the process involves an additional hop we need to update the service configuration, setting the logical address to the router address (the endpoint address above) and listentUri to our actual physical address. Once this was in place the message was received (shown with the output from our first subscriber):

To wrap things up I wanted to try sending a non-SOAP message via the router. I updated my router code to just create the router, commenting out the subscription to my WCF endpoint and the push of data to the router. Then using a seperate console application I used a HttpWebRequest to send a simple csv string:


string token = HttpGetAuthenticationToken("bitsinout", "********");
WebRequest req = HttpWebRequest.Create("http://bitsinout.servicebus.windows.net/productrouter/");            
req.Method = "POST";
req.Headers.Add("X-MS-Identity-Token", token);
req.ContentType = "text/csv";
StreamWriter sw = new StreamWriter(req.GetRequestStream());
sw.Write("12346,ProductB,350");

The string of data was successfully pushed through to my service. The HttpGetAuthenticationToken method is from the SDK samples and allows us to retrieve a temporary security token enabling requests to be authenticated on the Service Bus.

One thing to note, as the router only fowards messages, we need a separate router per data format required by each push subscriber and therefore the creation of these different formats would be down to the publisher. If a business has made the move to put pub/sub into the cloud, presumably they would also want message transformations there also.

.NET Services - Routers

noreply@blogger.com (Paul Arundel) — Wed, 22 Jul 2009 20:26:00 +0000

In my previous post I worked through a sample where a business could use the .NET services relay to expose an external service without making any infrastructure changes. For this post I will look at the how we can use the Service Bus Router to push data to external services.

My scenario is a simple push of product data updates to partners/customers who wish to subscribe to this information. With BizTalk I could create a simple solution where a product update message would be inserted into the BizTalk Message Box and send ports for each subscriber would be tailored to that subscribers message format and transport mechanism needs. Using .NET Services we will fire our product update into the cloud and allow the Service Bus to distribute the message to push subscribers. This offers a distinct advantage other than just moving the pub/sub infrastructure into the cloud. Partners who wish to receive this information can themselves use the .NET Services relay bindings to subscribe directly to messages sent to the router without having to expose a public service to the internet.

I'm going to look at a number of external partners as examples, each with different requirements in how they want the data pushed to them. In this post I will look at getting a message through to a partner who wants to connect directly to the router from an internal system using the relay bindings. I'll next look at how the router can push messages to a push subscriber web service endpoints that are not using relay bindings (i.e. just normal public web services that do not know messages are coming from the Service Bus). Finally I want to look at a push subscriber that cannot have a constant connection to the Service Bus and will instead use the Service Bus Queue functionality. My example for this last case would be a company that only wants to recieve product updates once a day as part of a batch process.

First the code for our message publisher. (Most of the code is based on/copied from the SoapRouter and HttpRouter samples):


Uri routerUri = ServiceBusEnvironment.CreateServiceUri("sb", "bitsinout", "productupdates");
TransportClientEndpointBehavior serviceBusCredential = GetCredential();
RouterClient client;
try
{
    client = RouterManagementClient.GetRouter(serviceBusCredential, routerUri);
    policy = client.GetPolicy();
}
catch (EndpointNotFoundException)
{
    policy = new RouterPolicy();
    policy.MessageDistribution = MessageDistributionPolicy.AllSubscribers;
    policy.Discoverability = DiscoverabilityPolicy.Public;
    policy.MaxSubscribers = 10;
    policy.Authorization = AuthorizationPolicy.NotRequired;
    client = RouterManagementClient.CreateRouter(serviceBusCredential, routerUri, policy);
}

ChannelFactory<IProductServicesChannel> factory = client.CreateRouterClient<IProductServicesChannel>(EndToEndSecurityMode.None, RelayClientAuthenticationType.RelayAccessToken);
factory.Endpoint.Behaviors.Add(serviceBusCredential);
IProductServicesChannel channel = factory.CreateChannel();
channel.Open();
channel.ProductUpdate(new Product() { Code = "12345", Name = "Meat Pie with Gravy", Price = 129 });

First we create our router uri, get the credentials allowing us to connect to the service bus and see if the router exists (RouterManagementClient.GetRouter()). If not(EndpointNotFoundException) we create the router using RouterManagementClient.CreateRouter() with an appropriate configuration (RouterPolicy). Once the router is established, we get a client of the router much like we would any other WCF service and call methods on this proxy (in our case channel.ProductUpdate()) to send messages to the router.

I've chosen the push subscriber relay scenario first as this is very easy to implement. On our subscriber client side we need the following code (this is
from the SoapRouter NetOnewaySubscriber project in the samples):


Uri routerUri = ServiceBusEnvironment.CreateServiceUri("sb", "bitsinout", "/productrouter/");
RouterClient client = RouterManagementClient.GetRouter(GetCredential(), routerUri);
ServiceHost host = new ServiceHost(typeof(ProductService));
ServiceEndpoint endpoint = client.AddRouterServiceEndpoint<IProductServices>(host);
host.Open();

That's it. The AddRouterServiceEndpoint does all the magic, creating a subscription to the router that lasts as long as the client is connected. This worked first time and messages were received by the client.

However, I was curious to see what happens when I run the publisher process a number of times. That is, what happens when I send a message to the router, close the channel, restart the process, get the router again and send another message, all while the push subscriber endpoint remains connected. This works fine, as you would expect, as the router would not have expired and still exist on the service bus. However, if I call client.DeleteRouter() after sending the first message (simulating an expired router) the subscriber endpoint remains connected, does not error and does not recieve the second message. Going further if I connect/subscribe the destination endpoint to the router before the router is created, it does not error but the publisher process that calls RouterManagementClient.CreateRouter() fails with an unhelpful error. Maybe I'm missing something (and .NET Services is not a finished product) but I suspect subscriber endpoints that connect using the relay bindings will have to be managed very carefully. I see little (or haven't found) much detail on error reporting. How do we know if a particular subscriber did not receive a message?

Pleased with my progress I've made an attempt to look at my next example, pushing data to external public web services. The SDK samples all use relay bindings for subscriber endpoints, presumably to avoid people trying the samples to worry about exposing a public web service (and tinkering with hardware router rules). Unfortunately in trying to get the Service Bus to push a message to public web service subscriber I've been halted by an error (maybe on my part) which I cannot work around and have sent feedback.

.NET Services - Integration Scenarios

noreply@blogger.com (Paul Arundel) — Tue, 07 Jul 2009 12:06:00 +0000

I experimented with the original BizTalk Services last year and created a toy application using the Multicast relay bindings. Now the renamed .NET Services appears to offer more features, particularly Routers and Queues, I would like to investigate further. I want to work through a few common business to business integration scenarios, where we might have used BizTalk or Web Services, and see how .NET Services can help. I don't expect any of this to be new, or even go further than the various SDK samples, but it's always worth working through a scenario to help understanding.

Let's start with a very simple example using the Service Bus as a relay: A company wishes to expose a product data service to the internet so customers can get the latest product pricing information. Normally this may involve infrastructure changes, new hardware and the time of the company network admins (e.g. an internet facing server in the DMZ or allowing specific inbound calls through the firewall). The firewall traversal functionality offered by the .NET Services Service Bus relay can give us this public endpoint without the infrastructure changes. I'm sure the company network admins would still like to have a say in this but lets plough on!

Before getting to the code it's worth noting Clemens Vasters other reasons for putting public services on the Service Bus relay:

In addition to providing NAT and Firewall traversal and discoverability the delegation of the public network endpoint into the Relay provides a service with a number of additional key advantages that are beneficial even if NAT traversal or discoverability are not a problem you need to solve:

The Relay functions as a "demilitarized zone" that is isolated from the service's environment and takes on all external network traffic, filtering out unwanted traffic.
The Relay anonymizes the listener and therefore effectively hides all details about the network location of the listener thus reducing the potential attack surface of the listening service to a minimum.
The Relay is integrated with the Access Control Service and can require clients to authenticate and be authorized at the Relay before they can connect through to the listening service. This authorization gate is enabled by default for all connections and can be selectively turned off if the application wants to perform its own authentication and authorization.

First off I've created a simple WCF service and WCF client, this could represent an already existing internal service at the company:


//Contract
[ServiceContract]
interface IProductService
{
    [OperationContract]
    Product GetProduct(string code);
}

//Service
ServiceHost host = new ServiceHost(typeof(ProductService));
host.Open();

//Client
ChannelFactory<IProductChannel> factory = new ChannelFactory<IProductChannel>("ProductService");
IProductChannel channel = factory.CreateChannel();
channel.Open();
Product prod = channel.GetProduct("ABC");
channel.Close();
factory.Close();
Console.WriteLine(prod.Name + " " + prod.Price);

And a couple of very simple configuration files to get things working using the standard basicHttp binding:



<service name="Arundel.NetServicesTest.Services.ProductService">
    <endpoint address="http://localhost/ProductService"
              binding="basicHttpBinding"
              contract="Arundel.NetServicesTest.Services.IProductService" />
</service>

<client>
    <endpoint name="ProductService"
              address="http://localhost/ProductService" 
              binding="basicHttpBinding"
              contract="Arundel.NetServicesTest.Services.IProductService" />
</client>

Now to make the changes to create a public endpoint. First I created a solution on my account at the .NET services Portal. All I needed here was a unique name for the relay address: sb://[solution name].servicebus.windows.net. Next I added an endpoint for the relay to my service:


<endpoint address="http://bitsinout.servicebus.windows.net/productservice"
    binding="basicHttpRelayBinding"
    contract="Arundel.NetServicesTest.Services.IProductService"
    behaviorConfiguration="RelayBehavior"/>

Three things have changed with this new endpoint. The address is now the public facing uri of the service, hosted on the Service Bus. The binding has been changed to the relay version of the basicHttp binding. Finally a new behavior was needed allowing the service to establish a connection with my service bus solution:


<behavior name="RelayBehavior">
    <transportClientEndpointBehavior credentialType="UserNamePassword">
        <clientCredentials>
            <userNamePassword userName="bitsinout" password="*********"/>
        </clientCredentials>
    </transportClientEndpointBehavior>
</behavior>

Ok first test was to update the client configuration to point at the service bus:


<client>
    <endpoint name="ProductService"
            address="http://bitsinout.servicebus.windows.net/productservice"
              binding="basicHttpRelayBinding"
              contract="Arundel.NetServicesTest.Services.IProductService"
              behaviorConfiguration="RelayBehavior"/>
</client>

Apart from the update of the address and binding you may notice that a behavior for authentication has been added as with the service. This was necessary otherwise a Cardspace window pops up when running the client as the relay bindings require authentication with the Service Bus solution. This was a good test of the relay but obviously consumers of the service will not be required to use a relay WCF binding and I would want my original basicHttp client configuration to stay the same apart from an updated address. I could only get this working by allowing anonymous access to the public endpoint (without delving too deeply into the security configuration options). Anonymous access can be enabled by adding the following to the service configuration:


<bindings>
    <basicHttpRelayBinding>
        <binding name="default">
            <security mode="None" relayClientAuthenticationType="None"/>
        </binding>
    </basicHttpRelayBinding>
</bindings>

Finally I wanted to test the service from a non WCF client. I therefore added an additional endpoint using the webHttpRelayBinding, first making a few changes to the service code switching from a ServiceHost to a WebServiceHost and added an appropriate WebGet attribute to the operation:


WebServiceHost host = new WebServiceHost(typeof(ProductService));

[OperationContract]
[WebGet(UriTemplate = "product/{code}")]
Product GetProduct(string code);

And the additional endpoint:


<endpoint address="http://bitsinout.servicebus.windows.net/products"
          binding="webHttpRelayBinding"
          contract="Arundel.NetServicesTest.Services.IProductService"
          bindingConfiguration="default"
          behaviorConfiguration="RelayBehavior"/>

<webHttpRelayBinding>
    <binding name="default">
        <security mode="None" relayClientAuthenticationType="None"/>
    </binding>
</webHttpRelayBinding>

Finally a simple http GET with a browser:

It's astonishing how easily WCF configuration and .NET services allow this to be implemented. However the ease of implementation would only be a small consideration, amongst many others, when a business decides to expose a public service in this way. My thoughts would be that .NET services could be very useful to medium sized organizations when establishing business to business integration processes with their partners/customers. It worth remembering that many business are still integrating by emailing excel spreadsheets to each other or FTPing large daily flat files. Any platform that could help the move away from those types of processes is a good thing.

Running BizUnit tests in Intellipad

noreply@blogger.com (Paul Arundel) — Thu, 18 Jun 2009 11:05:00 +0000

After a lot of work I've finally managed to get the tests running directly within Intellipad. My aim was to have a syntax highlighted BizUnitMgMode, where a key press would parse the source, start the BizUnit process and display the output in a new Intellipad buffer.

I initially decided to re-use the CodeDOM compiling part of my code and basically compile the input to a dll and then use a testing framework to run the tests all within the context of Intellipad. This didn't go well.

I tried MSTest first and thought it would be easy to hook up to the API and run the test programmatically. But there doesn't appear to be an API as most of the classes are internal. Then I thought to write a wrapper around MSTest.exe, but I couldn't see a way to get to the output as the exe presumably redirects the output to the test results files. Disappointing.

NUnit next. After looking through the NUnit source and learning more about NUnit than I wanted to, I had the test running within the Visual Studio debugger. But no luck running the process from within Intellipad. I gave up and decided to forget the assembly idea and use the BizTalk object model directly. I only needed to write a simple test wrapper around the test runs, count passes and failures and BizUnit does the rest.

The actual Intellipad mode was simple to set up using the code from the MUrl sample. The mode uses the GetParser() method directly from my compiler allowing Intellipad to recognise valid source files and syntax highlight. I set the Intellipad 'Extension' code attribute to recognize '.bzm' files and sure enough opening a valid .bzm file in Intellipad selects the correct BizUnitMgMode and syntax highlights nicely:

I bound a key press to run the tests and the BizUnit output scrolls by in the output window:

Oslo May CTP Updates

noreply@blogger.com (Paul Arundel) — Wed, 10 Jun 2009 10:09:00 +0000

I've installed the Oslo May CTP and updated my prototype Oslo BizUnit language to work with the various changes. I've only looked at the changes that affect my project and have not spent any time with the Repository, Quadrant or 'M'. Although in the latter case 'M' now refers to both the domain modeling language and the language for defining textual DSLs. So I am working with 'M' in the language definition sense. Clear?

The change in the meaning of 'M' has corresponding updates in the language tools. Project assembly references need to be updated from 'Microsoft.M.Grammar' to 'Microsoft.M'. The APIs have also changed. To compile your grammar and get a DyanamicParser the following changes are needed:


 /*OLD
MGrammarCompiler compiler = new MGrammarCompiler();
compiler.FileNames = new string[] { "../../Mg/BizUnit.mg" };
compiler.Compile(ErrorReporter.Standard);
                
DynamicParser parser = new DynamicParser();
compiler.LoadDynamicParser(parser);
return parser;
*/

//NEW
var options = new CompilerOptions() { 
    Sources = { new TextItem {
          Name = "../../Mg/BizUnit.mg",
          ContentType = TextItemType.MGrammar, 
          Reader = new StreamReader(syntax)}}};
                    
CompilationResults results = Compiler.Compile(options);
return results.ParserFactories["BizUnit.TestLanguage"].Create();

If you are using the Build task to compile your .mg file, the following change is needed:


//OLD
//return DynamicParser.LoadFromResource("BizUnitMg", "BizUnit.TestLanguage");

//NEW
return MImage.LoadFromResource(Assembly.GetExecutingAssembly(), "BizUnitMg.mx").
                  ParserFactories["BizUnit.TestLanguage"].Create();

You also need to update your .csproj file to use 'MCompile' rather than 'MgCompile' and the imported project from Microsoft.M.Grammar.targets to Microsoft.M.Embedded.targets.

The most significant API change for my project was the deprecation of the GraphBuilder class in favour of a new NodeGraphBuilder. NodeGraphBuilder itself will be removed at some point but represents a temporary bridge to work with until IGraphBuilder is removed.

NodeGraphBuilder provides a Node class to work with when processing a graph. This is an improvement on GraphBuilder where passing objects to various GraphBuilder methods was the only way to traverse and process the graph. Although NodeGraphBuilder provides a more natural interface, the graph structure has changed considerably. With GraphBuilder the graph of sequences and nodes was relatively easy to pick up, particularly as it closely resembled the graph view displayed by Intelipad. NodeGraphBuilder provides a considerably more complex model to work with, where the Node class is either Atomic (has a single value) or has a number of Edges which point to other nodes. This is described in detail in an article here by Clemens Szyperski.

I need to get a better understanding of the material covered in the article, but with a little patience I have managed to update my code to use NodeGraphBuilder. I spent a lot of time initially updating the projection of my syntax to a form that allowed me to make sense of the node graph. I particularly struggled with how my grammar related to the NodeKind concept and the graph patterns discussed in section 1.1.2 of the article. I gained some clarity when following section 1.1.1 of the aritcle and mapped branded nodes to the various tree node classes in the project (e.g. TestCaseNode or CopyFileNode) and used labels for the various atomic values (e.g. SourceFile, TimeToDelay).


protected TreeNode CreateTreeNode(Node n)
{
    string brand = n.Brand.Text;
    return BizUnitMgCompiler.CreateNode(brand, n);
}

protected string GetEdgeValueByLabel(Node n, string label)
{
    var e = n.Edges.Single(x => x.Label == label);
    if (e.Node.AtomicValue != null)
    {
        return e.Node.AtomicValue.ToString();
    }

    return String.Empty;
}

The CreateTreeNode helper method now uses the node brand to dynamically create the correct class. The second helper method retrieves a string value from a node's set of edges by label.

Finally Intelipad appears to run a little quicker, although there are still delays when editing a grammar and the graph view is rebuilt. This delay has improved from over a second to well under a second for me. A restart of Intelipad now and again seems to help.

Tree preview mode has disappeared. To get the previous input / grammar / graph split panel view you now have to:

Open your .mg file with Intelipad

Choose 'Split new input and output views' from the DSL menu

Select the input panel and open your input source file from the File menu.

In the mode menu of the input panel select '[name of your grammar].mg Mode'

In the mode menu of the output panel select '[name of your grammar].mg Output Mode'

Click the top left of your output panel and select the input source file.

MGrammar BizUnit DSL - overview of the compiler

noreply@blogger.com (Paul Arundel) — Sun, 31 May 2009 15:40:00 +0000

As mentioned in my previous post I wanted the compiler of my DSL for BizUnit to be able to generate different outputs. Initially this will either be an assembly that can be executed by MSTest/NUnit or BizUnit xml files for each of the tests. Once I have the compiler tidied up I want to look at directly executing the test scripts within Intellipad with an interactive mode like MUrl.

My compiler needs to be able to traverse the graph of the input file generated by the Oslo tools and create one of the two types of output. Taking the lead from Roger Alsing's excellent example post and dusting off my copy of the Design Patterns book, the Visitor pattern seems a good fit for the general design. This allows separation of the process to traverse our node graph from the actual operations we want to perform on each node. The operations being those that actually create output. Additionally each visitor is a separate class and can therefore build up appropriate state while each node is visited. The image below shows the base TreeNode class and a few of the classes for BizUnit steps:

Each node implements the Accept() method inherited from the abstract base class taking an instance of a visitor and calling the appropriate visitor method for the node type. Additionally each node class stores properties relevant to the BizUnit step it represents (e.g. The length of time to delay for the DelayStep) as the visitor will need access to these for the operations it will perform. Below is the code for a SequenceNode in the graph:


public class SequenceNode : TreeNode
{
    public SequenceNode(object node) : base(node) { }

    public override void Accept(IVisitor visitor)
    {
        foreach (object n in Builder.GetSequenceElements(Node))
        {
            TreeNode gn = CreateTreeNode(n);
            gn.Accept(visitor);

        }
        visitor.Visit(this);
    }
}

For the case of a SequenceNode, a System.DataFlow.GraphBuilder is used to retrieve all the elements in the sequence. The method CreateTreeNode() dynamically creates an instance of the appropriate node (by matching name to a node type) and calls Accept() on each node. An instance of the following class is created for a CopyFile:


public class CopyFileNode : TreeNode
{
    public string SourcePath { get; set; }
    public string CreationPath { get; set; }

    public CopyFileNode(object node)
        : base(node)
    {
        SourcePath = GetValue(node, "From");
        CreationPath = GetValue(node, "To");
    }

    public override void Accept(IVisitor visitor)
    {
        visitor.Visit(this);
    }

}

The GetValue() method is a helper method that uses a GraphBuilder to retrieve descendant nodes by label. In this case the CopyFile node will have descendant nodes named 'From' and 'To'. As there is no node processing needed the Accept() method simply calls the Visit() method on the visitor instance which can then perform the appropriate operations for a CopyFile.

I've created two implementations of the visitor interface to generate the two output types. The first uses the various classes in the System.Xml.Linq namespace to generate BizUnit xml files. The second the CodeDOM to generate a test assembly:

The visitor interface looks as follows (with a small number of BizUnit steps):


public interface IVisitor
{
    void Visit(BizUnitTestFixtureNode node);
    void Visit(TestCaseNode node);
    void Visit(VarNode node);
    void Visit(SequenceNode node);
    void Visit(BizUnitStepNode node);
    void Visit(CopyFileNode node);
    void Visit(DelayNode node);
    void Visit(ValidateFileNode node);
    void Visit(DBNonQueryNode node);
    void Visit(CleanUpNode node);
    void Visit(HttpPostNode node);
}

The following code illustrates how the visitor processes a DBNonQuery node and creates an xml element for the BizUnit step. The '_testDocs' variable is just a collection used to build up the various xml elements created as the process moves through the node tree.


public void Visit(DBNonQueryNode node)
{
    CreateTestStep(node, "BizUnit.DBExecuteNonQueryStep",
        new XElement("DelayBeforeExecution", 0),
        new XElement("ConnectionString", node.ConnectionStr),
        new XElement("NumberOfRowsAffected", node.NumberOfRowsAffected),
        new XElement("SQLQuery", new XElement("RawSQLQuery", node.Query)));
}


private void CreateTestStep(object node, string assemblyName, string typeName, params XElement[] xel)
{
    _testDocs.Last().Value.Descendants(_testStage).
        First().Add(CreateStepXElement("TestStep", assemblyName,typeName,xel));

}

private XElement CreateStepXElement(string name, string assemblyName, string typeName, params XElement[] xel)
{
    return new XElement(name, new XAttribute("assemblyPath", assemblyName), new XAttribute("typeName", typeName), xel);
}

As the visitor processes each node it adds a new XML element to the correct test stage of the current XDocument being worked on. Once all the nodes have been processed the XDocuments are saved to the file system. The CodeDOM visitor is very similar in process but obviously builds up the various code methods, attributes and expressions before saving as a .NET assembly.

Prototype of an MGrammar DSL for BizUnit

noreply@blogger.com (Paul Arundel) — Tue, 26 May 2009 20:34:00 +0000

Inspired by Yossi's Oslo based BizTalk deployment solution, I scanned around for a similar project to get experience of using the Oslo tools. I decided to try implementing a small language to write BizUnit tests.

While the declarative programming model used by BizUnit provides a lot of flexibility, the resulting XML test files are not the easiest to read, understand and maintain. I imagine the original intention was for the XML to be auto generated, but in my experience, most BizTalk devs are hand coding the xml test files. This makes it difficult to understand the test coverage of a solution's test scripts when presented with pages and pages of angle brackets.

So my aim is to write a simple DSL that will compile down to calling the various steps of the BizUnit framework and allow developers (and maybe non-technical testers) to easily write and understand tests for common integration scenarios. The language will use the same step approach as BizUnit but with a minimal syntax.

For example to copy a file in BizUnit you use the following XML:

<SourcePath>.\TestData\InDoc1.xml</SourcePath>

<CreationPath>.\Rec_01\InDoc1.xml</CreationPath>

</TestStep>

With the DSL we could simply write something like:

CopyFile ".\TestData\InDoc1.xml" To ".\Rec_01\InDoc1.xml"

This doesn't look a great deal, but a few lines of this language are much easier to understand than a few pages of the XML. Where possible I also think it makes sense to use sensible defaults for BizUnit step properties e.g.:

<SearchPattern>TransactionId_*.xml</SearchPattern>

<ComparisonDataPath>.\TestData\ResultDoc1.xml</ComparisonDataPath>

</ValidationStep>

</TestStep>

Using common defaults this can be reduced to:

ValidateFile "C:\Recv2\TransactionId_*.xml" With ".\TestData\ResultDoc1.xml"

Obviously the language can be extended to set the other properties, but my aim is to make the tests scripts as clean as possible.

To get started defining a language with Intellipad you create a blank .mg file for your MGrammar and a blank file for your input language, open the .mg file in Intellipad, select Tree Preview mode, select your blank input file and off you go. Once you get the basic idea of declaring tokens and syntax rules, Intellipad works its magic and starts syntax highlighting your input language on the left and displaying the resulting graph on the right. I've no real experience of creating DSLs, or compilers for that matter* but the few samples that come with Intellipad get you started surprisingly quickly.

After a few hours I had a working version of my language. The overall structure is similar to writing normal c# unit tests targeting MSTest or NUnit, but with each test case allowing a number of BizUnit steps and a cleanup step if needed. The screenshot below shows a couple of tests defined with Intellipad syntax highlighting:

I've allowed the use of variable definitions in the language. From my experience the paths to files or URLs are used repeatedly throughout BizUnit tests and variables allow these to be declared once.

Now I have the basic syntax of my language defined as an .mg file the Oslo tools allow me to create a compiled version of this syntax, an .mgx file. This in turn can be used to process any input files written in the language and produce an MGraph, a tree structure we can then parse to create the output we require. Intellipad dynamically shows a similar graph as you work on your DSL:

I will discuss my compiler as it stands in my next post. It is successfully compiling the syntax as defined and can output either a dll that can be executed directly (with NUnit/MSTest) or can generate matching BizUnit XML files. I think this latter option is important as any investment in time with this language will at least allow you to create standard BizUnit XML files should you decide not to use it fully.

Overall I'm not sure whether this adds a great deal to BizUnit. The syntax is much clearer, but it is another syntax to learn. You could also potentially write a few helper classes that wrap the BizUnitOM classes and allow you to write similar tests directly in c# avoiding this DSL malarkey completely.

The major issue though, is in limiting yourself to this easy syntax you're potentially losing BizUnit's major strengths, its flexibility and extensibility. Every BizTalk solution I have worked on has required custom BizUnit steps of some form. The language will need a way of expressing these custom steps cleanly.

* Apart from the obligatory BrainF*** to CIL compiler of course.