Clemens Vasters

SocketShifter - Network virtualization over the .NET Service Bus

Mon, 06 Jul 2009 16:06:48 GMT

Anyone using the .NET Service Bus should take a good look at the SocketShifter project started by Rob Blackwell and Richard Prodger from AWS in the UK. AWS stands for Active Web Solutions, not for the "other" AWS. The full project is up on Codeplex.

What makes SocketShifter significant is that it takes the network abstraction of SOAP, WS-Addressing, and the Service Bus full circle and layers the very bottom of that stack - plain TCP connections - as a virtualization on top of the the stack. In other words: SocketShifter allows you to create full-fidelity, bi-directional socket connections through the .NET Service Bus.

We've created something very similar to SocketShifter last year (we're using it for a few internal purposes), but haven't made it public so far. I'm glad that the AWS folks built this, so that you get to play with it.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 5: The Queue API for the rest of us

Tue, 07 Apr 2009 06:38:26 GMT

In Parts 3 and 4 of this series I’ve explained the REST protocol for the .NET Service Bus Queue capability. If that looked a little too complicated for your taste and you’d rather clean with a simple API surface, here’s the API that you’ll prefer.

We’ve got two simple classes in the SDK (in the Microsoft.ServiceBus assembly/namespace) that do the job. The QueueManagementClient allows you to create, renew and delete queues in/from the .NET Service Bus namespace:


            1: publicstaticclass QueueManagementClient 
            2: {

            3:    publicstatic QueueClient
CreateQueue(TransportClientEndpointBehavior credential, 
            4:                                          Uri queueUri, QueuePolicy queuePolicy);

            5:    publicstaticvoid DeleteQueue(TransportClientEndpointBehavior
credential, 
            6:                                   Uri queueUri);

            7:    publicstatic QueueClient
GetQueue(TransportClientEndpointBehavior credential, 
            8:                                       Uri queueUri);

            9:    publicstatic QueuePolicy
GetQueuePolicy(TransportClientEndpointBehavior credential, 
           10:                                             Uri queueUri);

           11:    publicstatic DateTime
RenewQueue(TransportClientEndpointBehavior credential, 
           12:                                      Uri queueUri, TimeSpan requestedExpiration);

           13: }

The most striking difference between this API (and the underlying SOAP protocol for which we still owe you some docs) and the REST API/Protocol is that there’s only a single URI to deal with. The WS-Transfer aligned protocol to manage the policy, the “Queue Transfer” protocol used to dequeue messages, and the “Enqueue” protocol to add messages to the queue are all overlaid over the exact same URI.

The only somewhat ugly element here is the TransportClientEndpointBehavior that’s our awkwardly named credentials container. That class was meant to evolve into something else and then we changed our mind at some point, which is how the baby got stuck with that name. We’ll give that a prettier moniker in one of the next CTPs and as part of an overhaul of the .NET Access Control integration into Service Bus.

The function of the methods themselves should be quite obvious. You can create a queue by applying a policy, you can delete a queue, attach to an existing queue, get the latest queue policy, and renew (extend the lifetime) of a queue. For completeness, here’s CreateQueue in context:


            1: Uri queueUri = ServiceBusEnvironment.CreateServiceUri("sb",
solutionName, "/MyQueue/"); 
            2: 

            3: QueuePolicy queuePolicy = new QueuePolicy(); 
            4: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

            5: 

            6: QueueClient client = QueueManagementClient.CreateQueue(credential, queueUri, queuePolicy);

            7: queuePolicy = client.GetPolicy(); //
get effective policy

The QueueClient class allows interaction with a Queue. QueueClient instances cannot be created directly, but must be created via the QueueManagementClient factory.


            1: publicsealedclass QueueClient 
            2: {

            3:     publicvoid DeleteLockedMessage(Message
message); 
            4:     publicvoid DeleteQueue(); 
            5:     public DateTime
GetExpiration(); 
            6:     public QueuePolicy
GetPolicy(); 
            7:     public Message
PeekLock(); 
            8:     public Message
PeekLock(TimeSpan timeout); 
            9:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages); 
           10:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages, TimeSpan timeout); 
           11:     publicvoid Purge(); 
           12:     publicvoid ReleaseLock(Message
message); 
           13:     public DateTime
Renew(TimeSpan requestedExpiration); 
           14:     public Message
Retrieve(); 
           15:     public Message
Retrieve(TimeSpan timeout); 
           16:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages); 
           17:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages, TimeSpan timeout); 
           18:     publicvoid Send(Message
message, TimeSpan timeout); 
           19:     public RouterSubscriptionClient
SubscribeToRouter(RouterClient routerClient, TimeSpan requestedTimeout); 
           20: }

As you can tell, the class is – well – a queue client. It’s using WCF’s Message class as its message abstraction and supports sending messages into the queue, reading messages off the queue in a destructive fashion (Retrieve), and reading messages off the queue using the Peek/Lock pattern which provides resilience against message loss if the message were to be lost in transfer or the receiver fumbled the message. What you’ll also notice is that there are RetrieveMultiple and PeekLockMultiple variants of the retrieval functions which allow for getting more data with fewer network roundtrips.

Again, there should be no surprises using the API. Here’s how you send:


            1: queueClient.Send(Message.CreateMessage(MessageVersion.Default, "Hello",
input), TimeSpan.MaxValue);

and here’s how you do a destructive read:


            1: Message message = queueClient.Retrieve();

and here’s how you use the Peek/Lock pattern:


            1: Message message = queueClient.PeekLock();

            2: string content
= message.GetBody<string>(); 
            3: queueClient.DeleteLockedMessage(message);

There are several examples in the SDK showing how to use the API for Queues. The OnewayQueueSender and SoapHttpQueueSender samples are particularly interesting since they just use WCF channels to enqueue, including all the bells and whistles you get from using a WCF channel. Here’s how the OnewayQueueSender does it:


            1: ChannelFactory<IOnewayChannel> channelFactory = 

            2:      new ChannelFactory<IOnewayChannel>(new NetOnewayRelayBinding(), 
            3:                                         new EndpointAddress(queueUri)); 
            4: channelFactory.Endpoint.Behaviors.Add(userNamePasswordServiceBusCredential);

            5: IOnewayChannel onewayChannel = channelFactory.CreateChannel();

            6: 

            7: string input
= Console.ReadLine(); 
            8: onewayChannel.Hello(new HelloMessage(input));

The logical next question is: So why is the Queue not hooked up to a WCF listener? Answer: That’s what we’ve got “buffered Routers” for. The queue very explicitly provides a ‘pull’ model and the WCF listener would abstract that away and turn it into ‘push’. Routers provide ‘push’ natively and they can embed a Queue. More on Routers in the next set of posts in this series.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 4: The REST Queue Protocol in Code Snippets

Tue, 07 Apr 2009 05:40:58 GMT

After I’ve gone through the dry facts of the REST Queue Protocol in Part 3 of this series, here’s some code to look at. The code snippets are ripped from the HTTP Queue sample we’ve got in the .NET Services SDK and you can get the copy/paste ready code from there.

The most straightforward way to acquire the required hashed security token (see my comments to Stefan Tilkov) using HttpWebRequest looks like this:


            1: staticstring HttpGetAuthenticationToken(string username, string password) 
            2: {

            3:     string tokenUri
= string.Format("https://{0}/issuetoken.aspx?u={1}&p={2}", 
            4:                                     ServiceBusEnvironment.DefaultIdentityHostName, 

            5:                                     username, Uri.EscapeDataString(password));

            6: 

            7:     HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUri);

            8:     using (var
tokenResponse = tokenRequest.GetResponse()) 
            9:     {

           10:         using (var
tokenResponseStream = tokenResponse.GetResponseStream()) 
           11:         {

           12:             byte[]
tokenBody = newbyte[500]; 
           13:             int tokenBodyLength
= tokenResponseStream.Read(tokenBody, 0, 500); 
           14:             return Encoding.UTF8.GetString(tokenBody,
0, tokenBodyLength); 
           15:         }

           16:     }

           17: }

           18:

Once you’ve got the token, you need to pick a place in the Service Bus namespace. We’ve got a built-in helper function in the SDK that sits on ServiceBusEnvironment and which knows all the right incantations:


            1:  Uri queueUri = ServiceBusEnvironment.CreateServiceUri(

            2:                    Uri.UriSchemeHttps, 

            3:                    solutionName, 

            4:                    "/MyHttpQueue/");

Invoked in this way, the method yields the URI https://yourproject.servicebus.windows.net/MyHttpQueue/ so there’s spectacularly little magic to it. The reason why we recommend that you use the method in .NET applications is that we’ve broken everyone’s apps going from the previous CTP to the current CTP due to the namespace restructuring and we’d like to avoid doing that again as we improve the namespace story. I don’t foresee any further change of the magnitude we had this time, however.

To create a new Queue, you first create a Queue policy and then apply it to the Service Bus namespace name. We’ve got a Queue Policy object in the SDK that I’ll use here to keep things short. I’m accepting all the defaults that are spelled out in Part 2 of the series, but explicitly I’m setting the expiration to 1 hour. The Queue Policy XSD schema in the the HTTP Queue sample’s readme document, by the way.


            1: QueuePolicy queuePolicy = new QueuePolicy(); 
            2: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

            3: queueManageUri = HttpCreateQueue(token, queueUri, queuePolicy);

The code for creating the Queue shouldn’t be very surprising. It’s a POST of an Atom 1.0 entry for which I’m using HttpWebRequest and the WCF 3.5 Syndication API. The SDK sample is using a rather terse version of the ‘correct’ way of doing things. I’m quoting the expanded variant from the Text140 sample that I showed at MIX’09:


            1: publicstatic QueuePolicy
HttpCreateQueue(string token, Uri queueUri, QueuePolicy
policy, 
            2:                                           out Uri
queueSelfUri, out Uri queueHeadUri) 
            3: {

            4:     HttpWebRequest webRequest;

            5:     HttpWebResponse webResponse;

            6: 

            7:     queueSelfUri = null; 
            8:     queueHeadUri = null; 
            9: 

           10:     //
Create a new syndication item and add the queue policy as an extension
           11:     SyndicationItem syndicationItem = new SyndicationItem(); 
           12:     syndicationItem.ElementExtensions.Add(

           13:         new SyndicationElementExtension("QueuePolicy", 
           14:                                         "http://schemas.microsoft.com/ws/2007/08/connect", 
           15:                                         policy));

           16: 

           17:     //
create a new POST request and set the Content-Type to represent an Atom 1.0 entry
           18:     webRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
           19:     webRequest.ConnectionGroupName = "queueclient"; 
           20:     webRequest.KeepAlive = false; 
           21:     webRequest.Method = "POST"; 
           22:     webRequest.Headers.Add("X-MS-Identity-Token",
token); 
           23:     webRequest.ContentType = "application/atom+xml;type=entry;charset=utf-8"; 
           24: 

           25:     //
write the item to the request stream
           26:     using (var
requestStream = webRequest.GetRequestStream()) 
           27:     {

           28:         using (var
textWriter = new XmlTextWriter(requestStream,
Encoding.UTF8)) 
           29:         {

           30:             syndicationItem.GetAtom10Formatter().WriteTo(textWriter);

           31:             textWriter.Flush();

           32:         }

           33:     }

           34: 

           35:     //
get the response
           36:     using (webResponse
= webRequest.GetResponse() as HttpWebResponse) 
           37:     {

           38:         return GetQueueInfoFromResponse(webResponse,
queueUri, out queueSelfUri, out queueHeadUri); 
           39:     }

           40: }

           41: 

           42: static QueuePolicy
GetQueueInfoFromResponse(HttpWebResponse webResponse, Uri queueUri, 
           43:                                                     out Uri
queueSelfUri, out Uri queueHeadUri) 
           44: {

           45:     queueHeadUri = null; 
           46:     queueSelfUri = null; 
           47: 

           48:     if (webResponse.ContentType.StartsWith("application/atom+xml;type=entry",
StringComparison.OrdinalIgnoreCase)) 
           49:     {

           50:         Atom10ItemFormatter atomItemFormatter = new Atom10ItemFormatter(); 
           51:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
           52:         {

           53:             atomItemFormatter.ReadFrom(responseReader);

           54:             //
we found it. let's get the policy and the queue head URI
           55:             foreach (SyndicationLink
queueSelfLink in
           56:                            from link in atomItemFormatter.Item.Links 
           57:                            where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
           58:                            select link)

           59:             {

           60:                 queueSelfUri = queueSelfLink.Uri;

           61:                 break; 
           62:             }

           63: 

           64:             foreach (SyndicationLink
queueHeadLink in
           65:                            from link in atomItemFormatter.Item.Links 
           66:                            where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
           67:                            select link)

           68:             {

           69:                 queueHeadUri = queueHeadLink.Uri;

           70:                 break; 
           71:             }

           72: 

           73:             return atomItemFormatter.Item.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
           74:                  "QueuePolicy", 
           75:                  "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
           76:         }

           77:     }

           78:     else
           79:     {

           80:         Atom10FeedFormatter atomFeedFormatter = new Atom10FeedFormatter(); 
           81:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
           82:         {

           83:             atomFeedFormatter.ReadFrom(responseReader);

           84: 

           85:             //
look for an item whose alternate-link equals the desired queue URI. The alternate 
           86:             //
URI is the 'tail' endpoint by which the queue is mapped into the namespace. We are 
           87:             //
comparing Authority and Path since the scheme might be http or https and we want to
match
           88:             //
either.
           89:             foreach (SyndicationItem
requestedItem in
           90:                          from item in atomFeedFormatter.Feed.Items 
           91:                          from link in item.Links 
           92:                          where 

           93:                            (link.RelationshipType.Equals("alternate",
StringComparison.OrdinalIgnoreCase) && 
           94:                             link.Uri.Authority.Equals(queueUri.Authority, StringComparison.OrdinalIgnoreCase) &&

           95:                             link.Uri.AbsolutePath.Equals(queueUri.AbsolutePath, StringComparison.OrdinalIgnoreCase)) ||

           96:                            (link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) && 
           97:                             link.Uri.Equals(queueUri))

           98:                          select item)

           99:             {

          100:                 //
we found it. let's get the policy and the queue head URI
          101:                 foreach (SyndicationLink
queueSelfLink in
          102:                                from link in requestedItem.Links 
          103:                                where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
          104:                                select link)

          105:                 {

          106:                     queueSelfUri = queueSelfLink.Uri;

          107:                     break; 
          108:                 }

          109: 

          110:                 foreach (SyndicationLink
queueHeadLink in
          111:                                from link in requestedItem.Links 
          112:                                where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
          113:                                select link)

          114:                 {

          115:                     queueHeadUri = queueHeadLink.Uri;

          116:                     break; 
          117:                 }

          118: 

          119:                 return requestedItem.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
          120:                      "QueuePolicy", 
          121:                      "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
          122:             }

          123:         }

          124:     }

          125:     returnnull; 
          126: }

The “magic” that isn’t straight up Atom Pub is in lines 10-15 above. I’m attaching the policy object to the SyndicationItem abstraction that will be written out as an Atom 1.0 entry. [I know that copy/paste is difficult using this format – I need to clean up that expanded utility class and will post it in downloadable form within the next few days]

The method returns the effective policy that the Service Bus is using, and yields the management URI for the queue where the Atom entry resides (self-link) as well as the head of the queue as output arguments. The queueUri to which you applied the entry has morphed into the Queue’s tail when the method returns successfully. Having the Queue by its tail, submitting messages into it is very simple. Just do a plain POST. Here we’re just writing a simple string from the local variable input as the entity body and send it.


            1: //
send
            2: HttpWebRequest sendRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
            3: sendRequest.Method = "POST"; 
            4: sendRequest.Headers.Add("X-MS-Identity-Token",
token); 
            5: sendRequest.ContentType = "text/plain;charset=utf-8"; 
            6: using (var
sendStream = sendRequest.GetRequestStream()) 
            7: {

            8:     using (var
writer = new StreamWriter(sendStream, Encoding.UTF8)) 
            9:     {

           10:         writer.Write(input);

           11:         writer.Flush();

           12:     }

           13: }

           14: sendRequest.GetResponse().Close();

For a simple destructive read, you use the ‘queue head’ URI and do a DELETE:


            1: HttpWebRequest dequeueRequest = 

            2:     HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

            3:                           "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
            4: dequeueRequest.ConnectionGroupName = "dequeue"; 
            5: dequeueRequest.Method = "DELETE"; 
            6: dequeueRequest.ContentLength = 0;

            7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
            8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
            9: {

           10:     if (response.StatusCode
== HttpStatusCode.OK) 
           11:     {

           12:         using (var
responseBody = response.GetResponseStream()) 
           13:         {

           14:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
           15:             {

           16:                 string data
= responseReader.ReadToEnd(); 
           17:                 Console.WriteLine(data);

           18:             }

           19:         }

           20:     }

           21: }

For a peek/lock read where you first lock the message and then delete the lock when you want to keep the message, you do a POST on the head and remember the X-MS-Message-Lock header value:


            1: HttpWebRequest dequeueRequest = 

            2:      HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

            3:                            "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
            4: dequeueRequest.ConnectionGroupName = "dequeue"; 
            5: dequeueRequest.Method = "POST"; 
            6: dequeueRequest.ContentLength = 0;

            7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
            8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
            9: {

           10:     if (response.StatusCode
== HttpStatusCode.OK) 
           11:     {

           12:         lockUri = new Uri(response.Headers["X-MS-Message-Lock"]); 
           13:         using (var
responseBody = response.GetResponseStream()) 
           14:         {

           15:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
           16:             {

           17:                 string data
= responseReader.ReadToEnd(); 
           18:                 Console.WriteLine(data);

           19:             }

           20:         }

           21:     }

           22: }

If you decide to keep the message (i.e. your app didn’t puke processing it), you DELETE the lock, otherwise you do a PUT. The code is practically the same except for the method so I’ll just quote DELETE:


            1: HttpWebRequest deleteLockedMessageRequest = HttpWebRequest.Create(lockUri) as HttpWebRequest; 
            2: deleteLockedMessageRequest.ConnectionGroupName = "lock"; 
            3: deleteLockedMessageRequest.Method = "DELETE"; 
            4: deleteLockedMessageRequest.KeepAlive=false; 
            5: deleteLockedMessageRequest.ContentLength = 0;

            6: deleteLockedMessageRequest.Headers.Add("X-MS-Identity-Token",
token); 
            7: deleteLockedMessageRequest.ContentLength = 0;

            8: deleteLockedMessageRequest.GetResponse().Close();

That’s it. Create, Enqueue, Destructive Read, Peek/Lock Read, and .. oh .. yes.. forgot Delete the Queue. Take the snippet above and swap out the lockUri for the queueSelfUri. Done.

Ah, yes, and please … don’t blame me for HTTP or HttpWebRequest or Clemens’ demo coding style requiring too many lines of code. I could have applied a bit of code-compression here, but I’m intentionally trying not to abstract away too much of the protocol here. I’m generally with you, though. The guys putting together the WCF REST Starter Kit are working on making code like that shrink.

The .NET Service Bus REST protocol for Queues - Some comments, some answers

Tue, 07 Apr 2009 04:14:16 GMT

Stefan Tilkov has several interesting remarks regarding our .NET Service Bus REST Queue Protocol that are worth addressing.

Putting a password in the URI to get an identity token seems to expose information unnecessarily

That’s an area where we know that we’re going to change the protocol. We’ve already labeled that protocol as temporary in the documentation for the PDC CTP and we didn’t get all the pieces in the .NET Access Control service together, yet. Since it’s a HTTPS call, the data doesn’t get exposed on the wire, though.

Queue creation seems fine, even though I feel a little uneasy about wrapping this in an Atom entry

Using Atom 1.0 and the Atom Publishing Protocol as the framework for managing the namespace is very intentional - for several reasons. First of all, it’s a standardized protocol for managing generic lists and the elements in those lists. With that we have a stable and accepted protocol framework and there’s plenty of tooling and framework support around it. That’s worth something. All we need to do is to add some simple extensions – the policies – on top of that stack. Beats having to define, version, and maintain a whole protocol.

On the other hand, Atom seems reasonable considering you get an Atom feed from the queue’s “parent” resource

That’s what I mean. All sorts of tools know how to navigate and display Atom.

Very nice to see the use of link/rel to get to the detailed Queue URIs; it would be even better if the rel values themselves were URIs in an MS namespace

I don’t see much potential for collision here and I would find it odd to have something as simple “self” and “alternate” and then add some unsightly QName for my rel expressions. Simple is good.

Using “alternate” for the tail seems strange

“self” refers to the location where the Atom <entry> resides. “alternate” is what the entry points to. Since the Queue gets mapped into the namespace by “sticking its tail out”, the choice of the alternate link is the simplest possible mapping I could think of.

“The queue’s tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.” The tail does not accept GET. Why not? Can’t think of a reason. Also, I don’t get why any method is allowed. It seems to me a POST/PUT combination or an idempotent POST variant would be the way to go.

The way to look at this is that the Queue’s tail is acting on behalf of the receiver/resource that’ll eventually pick up and process the messages. POST, PUT, DELETE, and BOOYAH are all operations that cleanly map to processing operations and can often be delivered asynchronously with a 202 receipt reply. GET and HEAD don’t make much sense when executed in an asynchronous fashion without getting a reply that’s backed by a response for the receiver. OPTIONS is simply reserved for future use.

“DELETE performing a destructive read” – huh? What happens if I don’t get back a response? How can I retry (as I can’t tell whether I get the same or a different message)? Maybe it’s intended to be used only if you don’t care about losing messages.

DELETE is indeed the dequeue operation variant that you’d use if you are ok with occasional message loss and want to trade transfer reliability for fewer roundtrips.

The POST lock/delete approach, on the other hand, is very nice. Maybe it should be made idempotent, again e.g. using POE

POST lock/delete is the dequeue variant where you are doing the trade the other way around. More reliability bought with an extra roundtrip. In my view, idempotent access to individual messages isn’t much of a practical priority for a competing consumer queue. You’ll get a message or a set of messages to look at under a lock and if you walk away from the message(s), those messages pop back into the queue for someone else to look at. There are obviously scenarios where you want to look at a message sequence in other ways than an ordered queue where you can only get at messages as they appear on the head of the sequence – direct message access and per-message idempotent access matter in those scenarios and we’re looking to give you a capability of that sort in a different kind of messaging primitive.

“The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0)”; although my immediate reaction was to question whether DELETE ever carries a body, the HTTP spec indeed doesn’t say anything about this

One of the great things about working here is that there are all sorts of interesting people around. I’ve discussed the use of DELETE and whether you can provide an entity body in either direction with Henrik Frystyk Nielsen, who works as an architect on the WCF team and is one of the co-authors of HTTP 1.1. Henrik’s stance is that all operations allow entity-bodies unless it’s explicitly forbidden in the spec. I don’t have a better authority to talk to.

“The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue’s head URI” - the wording is worse than the actual approach.

I’m sorry that my writing is so clumsy ;-)

Hang In There. Not Done With Explaining Routers and Queues, Yet :)

Tue, 07 Apr 2009 03:20:24 GMT

I’m far from being done with my blog series on Queues and Routers. Here’s how far I’ve yet gotten: Part 1 – Fundamentals, Part 2 - Queue Policies, Part 3 - Queue REST Protocol. I still owe you the Queue API, the Router Policies, the Router REST Protocol, and the Router API. It’s simply that the planning work for the next milestone slows me down a bit - sorry. If you also want the smaller bits of info, start following me on Twitter in the meantime.

.NET Services March 2009 CTP: Host a Public Website At The Kitchen Table or from a Coffee Shop! No Kidding.

Sun, 05 Apr 2009 19:41:36 GMT

.NET Service Bus Reverse Web Proxy: Click here to download the source

Using the application/service built from the sample linked at the top of this post you can host a publicly discoverable and accessible website or Web service from your Windows notebook or desktop machine from within most network environments without having to open up a port on the firewall, mapping a port on your NAT, or using some type of dynamic DNS service to make the site discoverable. All those essential connectivity features are provided by the .NET Service Bus and with the help of the included sample code.

I’m intentionally not bundling this up as a conveniently installable binary along with a nice configuration UI – that’s not my role here. If you want to grab the code and make it part of a cool personal media sharing app, provide external access to a departmental enterprise app, put a prototype out there for a client to play with, host a web service you want to show off, or or provide an installable version with a nice configuration UI – go ahead.

The attached sample application/service has two key capabilities that I’ve repeatedly been asked for:

a) It is a reverse web proxy that can run either as a console application or as a Windows (NT-) service. The reverse web proxy can sit in front of any web server and forward requests to it. I’ve tested this only with IIS as the backend, but I don’t see a reason why this shouldn’t work with Apache or the Web Server built into some J2EE application server.

b) It is a scripting policy host that projects the crossdomain.xml and ClientPolicyAccess.xml files required by Adobe Flash and Microsoft Silverlight into the root of a .NET Services namespace, permitting cross-domain script access from Flash and Silverlight for all endpoints hosted within the namespace. You can easily adjust the code in the sample to restrict access to particular resources within the namespace.

The fundamental architecture is illustrated in the picture. The web application that you want to project out to the public internet sits on some web server on your machine. “Your machine” may be a desktop machine at home or at work or a notebook in a hotel lobby or an airport on WiFi. As long as you’ve got line-of-sight to the .NET Service Bus and the TCP ports 828 and 818 are available for outbound traffic, you’re good. The reverse web proxy app will map any local HTTP server to a name in the .NET Service Bus and forward the traffic between the .NET Service Bus and the HTTP server. The client (any web browser, but also any HTTP Web Service client) will talk to the .NET Service Bus at the given name, the traffic flows to the reverse proxy on your machine and from there to the HTTP server.

I’m hosting (for a few days) a sample dasBlog site instance at http://clemensv6.servicebus.windows.net/dasblog/. The hosting machine for that blog is one of my personal machines. It’s got a local network address assigned by DHCP, it’s not listed in any NAT mappings, and it’s local Firewall isn’t even open for inbound HTTP traffic.

How to install, build, and run

As a prerequisite you will need three things:

Visual Studio 2008 SP1 with the .NET Framework 3.5 SP1.
A .NET Services project account. The quickest route is to go to http://portal.ex.azure.microsoft.com and click “Sign up”. The approval/provisioning is pretty much instantaneous (plus 20 seconds for the provisioning to run through) once you provide your Windows Live ID. No more access codes.
The .NET Services SDK for the March 2009 CTP. Click here to get it.

Unpack the files, and open ServiceBusReverseWebProxy.sln with Visual Studio 2008. In the ServiceBusReverseWebProxy project, find the app.config file and open it. Here’s where you need to put your project name and password and where you map your sites:


            1: <?xml
version="1.0" encoding="utf-8" ?>
            2: <configuration>
            3:     <configSections>
            4:         <sectionname="reverseWebProxy"
            5:                  type="Microsoft.Samples.ServiceBusReverseWebProxy.ReverseWebProxySection,
ServiceBusReverseWebProxy"/>
            6:     </configSections>
            7:     <!--
Add your .NET Services project account information here -->
            8:     <!--
Create a project at http://portal.ex.azure.microsoft.com/ -->
            9:     <reverseWebProxynetServicesProjectName="!!myproject!!"netServicesProjectPassword="!!mypassword!!"
           10:                      enableSilverlightPolicy="true">
           11:     <pathMappings>
           12:       <addnamespacePath="mysite"localUri="http://localhost/mysite/"/>
           13:     </pathMappings>
           14:   </reverseWebProxy>
           15: </configuration>

Put your .NET Services project/solution name into the netServicesProjectName and the password into netServicesProjectPassword.

Then pick a local HTTP server or site and give it a name in your .NET Service Bus namespace. That mapping is done in the <pathMappings> section. There are a few things that are important to note here:

If your project name were ‘clemensv6’ and you map some local URI to the namespacePath ‘dasBlog’, the resulting .NET Service Bus URI would be http://clemensv6.servicebus.windows.net/dasblog/.
The web application should only emit relative paths for links or, otherwise, should have a way to specify the external host address for links. That means that the web application needs to be able to deal with the presence of a reverse proxy. There is no content-level URL rewriter in this sample that would make any corrections to HTML or XML that’s handed upstream. DasBlog allows you to specify the blog site address as some external address and therefore satisfies that requirements.
Redirects and any other HTTP responses that emit the HTTP ‘Location’ header or any other HTTP headers where URIs are returned are rewritten to map the internal view to the external view.
If you set enableSilverlightPolicy to true, there will be crossdomain.xml and ClientPolicyAccess.xml endpoints projected into the root of your project’s namespace, ie. http://clemensv6.servicebus.windows.net/crossdomain.xml

Build. Run.

By default, the ServiceBusReverseWebProxy.exe application will simply run as a console application. If you use installutil –i ServiceBusReverseWebProxy.exe the application will be installed as a Windows Service. The default identity that it is installed under is ‘NETWORK SERVICE’. In restricted networks with constrained IPSec policies (such as the Microsoft Corporate Network), you may have to use a user account instead. You may also have to use some special Firewall-gateway software such as the ISA Firewall client to allow for outbound access to ports 828 and 818.

The actual application code isn’t really all that complicated. The ‘beef’ is in ReverseWebProxy.cs. What might be surprising here is that this class doesn’t use the WCF Service Model, but is using naked WCF channels for the upstream traffic to .NET Services and it’s using HttpWebRequest for the downstream traffic to the local Web Server. The reason for using channels is that the app is never doing any processing on the messages, so the channel model is the most straightforward and efficient way. The reason for using HttpWebRequest is that you can’t suppress auto-redirects on a WCF HTTP client. Since the stack needs to be completely transparent to redirects so that it’s the browser client up on top that gets redirected instead of someone on the way, I simply couldn’t use a WCF channel downstream. Seems to be one of these edge cases that the WCF team downstairs didn’t think anyone would ever need.

Let me know whether and how this works for you. Share the code, improve it, re-blog, let me know. @clemensv on Twitter, same name @microsoft.com for email.

XML-RPC with WCF (Updated)

Fri, 03 Apr 2009 17:09:14 GMT

XML-RPC for WCF Download here

I had updated my WCF XML-RPC stack for PDC’08 but never got around to post it (either too busy or too lazy when not busy). The updated source code is attached to this post.

Contrary to the code that I’ve posted a while back, the new XML-RPC implementation is no longer a binding with a special encoder, but is implemented entirely as a set of behaviors and extensions for the WCF Service Model. The behavior will work with WCF 3.5 as it ships in the framework and also with the .NET Service Bus March 2009 CTP.

The resulting Service Model programming experience is completely "normal". That means you can also expose the XML-RPC contracts as SOAP endpoints with all the advanced WCF bindings and features if you like. The behaviors support client and service side. I stripped the config support from this version – I’ll add that back once I get around to it. Here's a snippet from the MetaWeblog contract:


            1: [ServiceContract(Namespace = http://www.xmlrpc.com/metaWeblogApi)]
            2: publicinterface IMetaWeblog
: IBlogger 
            3: {

            4:    [OperationContract(Action="metaWeblog.editPost")] 
            5:    bool metaweblog_editPost(string postid, 
            6:                              string username, 
            7:                              string password, 
            8:                              Post post,

            9:                              bool publish); 
           10: 

           11:    [OperationContract(Action="metaWeblog.getCategories")] 
           12:    CategoryInfo[] metaweblog_getCategories( string blogid, 
           13:                                             string username, 
           14:                                             string password); 
           15:     ...

           16: 

           17: }

Setting up the endpoint is very easy. Pick the WebHttpBinding (or the WebHttpRelayBinding for .NET Service Bus), create an endpoint, add the XmlRpcEndpointBehavior to the endpoint and you’re good to go.


            1: Uri baseAddress = new UriBuilder(Uri.UriSchemeHttp,
Environment.MachineName, -1, "/blogdemo/").Uri; 
            2: 

            3: ServiceHost serviceHost = new ServiceHost(typeof(BloggerAPI)); 
            4: var epXmlRpc = serviceHost.AddServiceEndpoint(

            5:                   typeof(IBloggerAPI), 
            6:                   new WebHttpBinding(WebHttpSecurityMode.None), 
            7:                   new Uri(baseAddress,
"./blogger")); 
            8: epXmlRpc.Behaviors.Add(new XmlRpcEndpointBehavior());

The client is just as simple:


            1: Uri blogAddress = new UriBuilder(Uri.UriSchemeHttp,
Environment.MachineName, -1, "/blogdemo/blogger").Uri; 
            2:             

            3: ChannelFactory<IBloggerAPI> bloggerAPIFactory = 

            4:      new ChannelFactory<IBloggerAPI>( 
            5:              new WebHttpBinding(WebHttpSecurityMode.None), 
            6:              new EndpointAddress(blogAddress)); 
            7: bloggerAPIFactory.Endpoint.Behaviors.Add(new XmlRpcEndpointBehavior()); 
            8: 

            9: IBloggerAPI bloggerAPI = bloggerAPIFactory.CreateChannel();

           10:

For your convenience I've included complete Blogger, MetaWeblog, and MovableType API contracts along with the respective data types in the test applications. The test app is a small in-memory blog that you can use with the blogging function of Word 2007 or Windows Live Writer or some other blogging client for testing.

Of the other interesting XML-RPC APIs, the Pingback API has the following contract:


            1:  [ServiceContract(Namespace="http://www.hixie.ch/specs/pingback/pingback")] 
            2:  publicinterface IPingback 
            3:  {

            4:      [OperationContract(Action="pingback.ping")] 
            5:      string ping(string sourceUri, string targetUri); 
            6:  }

and the WeblogUpdates API looks like this:


              1:     [DataContract]
              2:     publicstruct WeblogUpdatesReply 
              3:     {

              4:         [DataMember]

              5:         publicbool flerror; 
              6:         [DataMember]

              7:         publicstring message; 
              8:     }

              9: 

             10:     [ServiceContract]

             11:     publicinterface IWeblogUpdates 
             12:     {

             13:         [OperationContract(Action = "weblogUpdates.extendedPing")] 
             14:         WeblogUpdatesReply ExtendedPing(string weblogName, string weblogUrl, string checkUrl, string rssUrl); 
             15:         [OperationContract(Action="weblogUpdates.ping")] 
             16:         WeblogUpdatesReply Ping(string weblogName, string weblogUrl); 
             17:     }

The code is subject to the Microsoft samples license, which means that you can freely put it into your (blogging) apps as long as you keep the house out of trouble.

How do I use the .NET Services "December 2008" CTP SDK with the "March 2009" CTP Services?

Fri, 03 Apr 2009 00:36:18 GMT

Short answer: You can’t.

There is a range of breaking protocol changes between the December bits and the March bits. Your app won’t work until you upgrade (uninstall/install) to the March 2009 CTP client bits and/or SDK.

.NET Services March 2009 CTP - Service Bus Routers And Queues - Part 3: The REST Queue Protocol

Thu, 02 Apr 2009 22:08:13 GMT

[If this text looks vaguely familiar you’ve read the HttpQueueSample Readme doc from the SDK. Good job!]

Here is the Service Bus Queue REST protocol. I apologize if this is a bit dry, but I want to give you the bare protocol facts first and will follow-up with code samples later. Look at the HTTP Queue SDK sample if you can’t wait for code ;-)

You can find the HTTP Queue sample in the SDK install path under .\Samples\ServiceBus\ExploringFeatures\Queues\HttpMessages\

The beauty of REST is that there’s a uniform interface. It’s all just GET, POST, PUT, DELETE, a set of resources, and a rigorous application of the RFC2616 / HTTP 1.1 rules. That doesn’t make the description particularly entertaining, but it does yield a pretty consistent interaction model:

Authorization

All Queue operations require the client to be appropriately authorized. The client must therefore acquire an lightweight identity token from the Microsoft .NET Access Control Service using the .NET Services solution name and the solution password and include the acquired token with each request. The tokens are added to requests using the X-MS-Identity-Token HTTP header. The value for this header is a short-lived token which can be used for a period of up to 8 hours and must be acquired from the .NET Access Control Service. We strongly encourage you to guard the acquired tokens in the same fashion as you would guard a credential and you should not openly expose them unprotected on the network or web-pages or embedded in a Flash or Silverlight application.

The request for acquiring the token is an HTTPS GET on the URI https://accesscontrol.windows.net/issuetoken.aspx?u={solutionName}&p={solutionPassword} whereby you replace the arguments for solution name and the solution password with your .NET Services solution credentials. If a token is successfully issued to you the request returns with a 200 (OK) status code and contains a text/plain entity body with a short, base64 encoded token hash as a single line of text, which you for the value of the X-MS-Identity-Token header.

Creating a Queue

A Queue is created on the Service Bus in four simple steps. I’ve discussed most of this in the post on Queue Policies:

Select a name in the Service Bus naming hierarchy where the Queue should be located, i.e. https://mysolution.servicebus.windows.net/myapp/q1
Create a Queue policy and define the desired properties of the Queue.
Embed the Queue policy into an ATOM 1.0 <atom:entry> as an extension and POST the entry to the designated Queue URI with the content-type application/atom+xml;type=entry.The request must carry an X-MS-Identity-Token header and the respective identity must have 'Manage' permission on for scope covering the Queue URI.
If the queue was successfully created, the POST request returns with a 201 'created' status code along with a Location header. The response entity body contains the effective Queue policy as applied to the name by the Service Bus.

The location header contains the queue's management URI that you need to retain in your application state to have access to the queue's metadata and management functions. If the queue could not be created successfully, the request may yield the following status codes:
- 400 - Bad Request. The policy was malformed or invalid.
- 403 - Forbidden. The client did not provide a X-MS-Identity-Token header, the provided token is no longer valid for use, or the provided identity is not authorized to create a Queue at this location.
- 409 - Conflict. There is already a Queue with an incompatible policy at the given location or the location is occupied by a Router or a Service Bus listener.
- 415 - Unsupported Media Type. The request did not carry the required content-type header.
- 500 - Internal Server Error. The processing failed to to a condition internal to the Service Bus service.

REST Anatomy of a Queue

Each Queue has five distinct types of resources: Policy, Tail, Head, Locks, and Control. The concrete URIs are discoverable using the Service Registry's Atom Publishing protocol. you should not make any assumptions about the particular format of the URIs spelled out below since the particular format might change.

The Atom 1.0 entry shown below is also enclosed in the response entity body of the create request (POST) explained above. A Queue's representation in the registry feed as seen when doing a GET (discovery-) request on the queue's parent URI is commonly as follows:

<feed xmlns="http://www.w3.org/2005/Atom">
  ...
  <entry>
    <id>{id}</id>
    <title type="text">MyHttpQueue</title>
    <updated>{date}</updated>
    <link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />
    <link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />
    <link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />
    <link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />
    <QueuePolicy xmlns="http://schemas.microsoft.com/ws/2007/08/connect">
      <Discoverability>Public</Discoverability>
      <ExpirationInstant>{expiration}</ExpirationInstant>
    </QueuePolicy>
  </entry>
</feed>

Managing The Queue

<link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />

The "self" link in the entry above is the queue's management URI that allows you to interact with the queue's policy.

To renew a queue and extend its expiration you PUT an updated Atom 1.0 entry with the effective policy and an updated ExpirationInstant value to the "self" URI. The "self" URI is the same URI as the one returned in the Location header returned by the create POST explained above. The request will yield a 200 status code if the renewal is successful. It will yield a 404 if the queue no longer exists.
To delete a queue you issue a DELETE request on the queue's management URI. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the queue was deleted successfully.

Enqueue

<link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />

The "alternate" link refers to the tail of the queue. The Queue's tail resource is where senders submit messages into the Queue. The queue's tail URI is identical to the URI where the queue was initially created. The queue's tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.

The endpoint should be considered a 'delegate' of the message recipient and not the resource that the respective operations have an immediate effect on. That means that the HTTP method applied to the Queue tail doesn’t have any particular semantic. If a message has been successfully accepted into the queue, the request returns a 202 status code. Otherwise it will return a 500

Dequeue and Peek/Lock

<link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />

The "queuehead" link refers to the head of the Queue. The Queue's head resource is where receivers retrieve messages from the queue. It is modeled as a resource whose current representation corresponds to the message(s) that reside at the head of the queue and which is/are next in line to be retrieved by consumers. Permitted operations on the head are:

DELETE: Delete performs a destructive read on the queue whereby the message(s) at the head of the queue are permanently deleted and returned as the entity body of the response to the request. The content type and formatting of the response depends on a set of query parameters that are discussed below. The DELETE request's response status code is 200 if at least one message could be retrieved and deleted from the queue and the message content is enclosed in the response's entity body. The response status code is 204if no message could be retrieved/deleted.
POST: Post creates a lock on the message(s) at the head of the queue and returns the locked messages. Locked messages are temporarily removed from the head of the queue for a period of 1 minute during which time the receiver can decide whether to return the message to the head of the queue or whether to permanently delete it. The POST request's response status code is 200 if at least one message could be retrieved and locked on the queue and the message content is enclosed in the response's entity body. The response status code is 204if no message could be retrieved/locked.

Releasing and Deleting Peek/Lock Message Locks

Any retrieved, peek/locked message (POST on “queuehead”) contains an HTTP header X-MS-Message-Lock whose value is a URI. The following operations can be performed on this URI:

DELETE: The message has been accepted/processed by the receiver and the lock shall be deleted. The deletion of the lock discards the message permanently from the queue. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was deleted successfully.
PUT: The message has been not been accepted/processed by the receiver and the message shall be put back at the head of the queue. This operation transitions the lock from the locked state into the unlocked state which releases the message back into the queue. The lock itself is discarded afterwards. The Put request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was released successfully.

Options for Dequeue and Peek/Lock

The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue's head URI. The options are the same for both operations:

timeout= value - The value is a numerical value (expressed in seconds) that indicates how long the client is willing to wait for the polling request to complete. The value should not exceed 60 seconds and a value of 30 seconds is a safer choice with intermediate proxies in place. If the timeout expires and no message is available, the request completes with a 204 response code. The default value is zero (0), which means that the request returns immediately.
maxmessages=value - Indicates how many messages the client is willing to accept in a single response. The default is 1. The service will return at most 10 messages at a time and a value greater than zero requires encoding=multipart (see below)
encoding=value - There are three supported encoding modes for the REST retrieval model
- asreply - The request stored in the Queue is mapped onto the response to the DELETE/POST request returning the message from the queue. This option requires maxmessages=1 or omission of the maxmessages option.
- single - The request stored in the queue is returned as a complete HTTP request frame on the response to the DELETE/POST request using Content-Type application/http. This option requires maxmessages=1 or omission of the maxmessages option.
- multipart - The request(s) stored in the queue are returned as a complete HTTP request frames on the response to the DELETE/POST request using Content-Type multipart/mixed with MIME parts of Content-Type application/http.

Queue Content Control

<link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />

The "queuecontrol" link refers to the control resource provides information about the status of the queue and allows for purging all of the queue's contents. The control resource is present in the current CTP, but you cannot perform any operations on it.

.NET Services March 2009 CTP - Service Bus Routers And Queues - Part 2: Queue Policies

Thu, 02 Apr 2009 20:10:25 GMT

In the previous post in this series I’ve discussed some of the foundational principles of the new Queue and Router features of the .NET Services Bus and the role that policies play in turning names in the namespace into messaging primitives. In this post I’ll start drilling into the capabilities of the Queue feature and will discuss what’s in the queue policy.

The capabilities of the Queue are best explored by looking at the Queue policy. Policies can be expressed in two ways: You can either you use the .NET Services SDK’s Microsoft.ServiceBus.QueuePolicy class with a .NET application or you can just whip up a bit of XML. The structure and the serialized representation of that class is exactly equivalent to an XML queue policy you’d wire up “by hand” and thus I’ll discuss the policy in terms of its XML elements:

The queue policy has the element name “QueuePolicy” within the XML namespace “http://schemas.microsoft.com/ws/2007/08/connect”. That’s the namespace we use for most protocols and data structures in the .NET Service Bus at this development stage and you should expect that we’re migrating to the “final” V1 namespaces in one of the next CTPs. If you use the .NET class you’ll just have to recompile to snap to new namespaces once they come around.

Below is a list of the policy’s elements and the permitted values. All elements are optional and can appear at most once. The default value applies when an element is omitted. It’s perfectly ok to send an empty QueuePolicy and accept the default values to keep things simple. I’d recommend to always set the ExpirationInstant value, however.

One required forward reference: Don’t get confused or distracted trying to figure out from here what it means “to provide an authorization token”. The docs explain this and I’ll also address this in the next blog post when I dive into the protocol.

Authorization – The authorization setting indicates whether sender and/or consumers must provide a security token when interacting with the queue. You can explicitly permit the .NET Service Bus to allow anonymous consumers and/or anonymous senders on a Queue. If a token is required, a sender must present a token with “Send” permission that is issued by the .NET Access Control service and the consumer must present a token with “Listen” permission that is issued by the .NET Access Control service. The respective rules are set up in the .NET Access Control service on your project’s “Service Bus” scope. Mind that all anonymous traffic is attributed to the “manager” of a Queue, i.e. to whoever sets the policy; all traffic is metered and accounted for. The permitted values for this element are:
- Required – (default) Senders and Consumers must provide an authorization token to send/retrieve messages.
- RequiredToSend – Senders must provide an authorization token with a “Send” claim to send messages, consumers don’t.
- RequiredToReceive – Consumers must provide an authorization token with a “Listen” claim to receiver messages, senders don’t.
- NotRequired – Neither senders nor consumers must provide a token, i.e. the queue is set up for anonymous traffic.
Discoverability – The discoverability setting defines under which conditions the Queue and its policy are visible in the Service Registry Atom feed and any forthcoming discoverability mechanisms. When you are browsing the Service Registry feed from within a browser, the only setting that will make the Queue visible is Public since the browser isn’t able to attach an authorization token without some scripting assistance.
- Managers - (default) The Queue and its policy are only discoverable if you provide an authorization token with the request and the token carries a “Manage” claim.
- ManagersListeners – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” claim.
- ManagersListenersSenders – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” and/or “Send” claim.
- Public – The Queue and its policy are discoverable without any authorization requirement, i.e. the general public.
ExpirationInstant – This is an XML dateTime value indicating the TTL (time-to-live) for the Queue. You can make Queues long-lived or short-lived. A Queue’s lifetime may be extended by changing this policy value and reapplying the policy. Once a Queue expires it is removed from the system along with all the messages that reside in it. The system may limit the Queue lifetime in the effective policy that you get back from the create/update request, but we’ll typically allow at least 24 hours without renewal. This value has a default of 24 hours, a maximum of 21 days and a minimum of 30 seconds. The value must represent a future dateTime and must be expressed in UTC.
MaxMessageSize – This is a numeric value defining the maximum size of any individual message that can be accepted into the Queue. The maximum message size is expressed in bytes and includes all infrastructure-imposed and encoding-related overhead. The actual overhead varies significantly based on whether messages are sent as SOAP 1.1, SOAP 1.2, or plain HTTP frames using text or binary encoding and whether a security token is required. A very defensive assumption is to reserve 10-12KB for protocol overhead in complex cases with WS* Security; the minimal allocation for protocol overhead should be around 4KB. The default and maximum values for the maximum message size is 60KB (60*1024 bytes). The minimum value is 8KB. We suggest that the payload size for an individual message does not exceed48KB, even though you can try to push it a bit. The expectation is that these values will trend up but I don’t see them more than doubling due to throughput, timeliness and scale considerations. If your data is larger you should consider how you can chunk it up.
TransportProtection – The transport protection setting defines whether senders and consumers must use HTTPS to interact with the Queue. This is the default setting. You would modify this to accommodate clients – typically on devices - that don’t do HTTPS all that well. Permitted values:
- AllPaths - (default) Any interaction with the queue requires HTTPS.
- None – Any interaction may be performed using either HTTP or HTTPS.
EnqueueTimeout – The enqueue timeout is an XML duration value that specifies how long an enqueue operation will hang if the queue is at capacity (full). The default value is 10 seconds, the minimum is 0 seconds, and the maximum is 60 seconds. If the timeout expires and the message could not be added to the queue during that time, the queue will act according to the Overflow policy setting.
MaxConcurrentReaders - [this setting not supported/enforced in the March 2009 CTP] The maximum concurrent readers value defines how many concurrent readers are permitted on the queue. If this numeric value is smaller than the default value of 2^31 (max int), the queue protocol will switch into ‘session mode’ that grants a limited number of read-locks on the queue. The minimum value is 1.
MaxDequeueRetries - [this setting not supported/enforced in the March 2009 CTP] The maximum dequeue retries value defines how often a message may be peek/locked and put back into the queue until it is considered poisonous, i.e. after how many retries it should be expected the the consumer will not be able to ever consume the message successfully, because it is malformed or the consumer experiences an error condition that requires some form of manual intervention (including a bug fix). If the message is found to be poisonous it will be sent once and without any retries to the endpoint specified in the policy’s PoisonMessageDrop value.
MaxMessageAge – The maximum message age is an XML duration value indicating after what time any enqueued message is considered ‘stale” and will be automatically dropped and removed from the queue. The default value is 10 minutes (600 seconds), the minimum value is 0 seconds (which effectively means that all incoming messages get dropped), the maximum value is 7 days.
MaxQueueCapacity – This numeric value indicates the maximum size of the Queue in bytes. This is a system-calculated value that cannot be set on the .NET class and should not be set by clients creating policies from scratch. The default and maximum capacity of any queue is currently capped at 2MB. This is a limitation specific to the March 2009 CTP , it’s been painful to impose this constraint, and it’s absolutely expected that this limit will be expanded significantly. 2MB still allow for a several hundred notification messages of a 2-4KB. If you need to store more data you can absolutely create several queues and partition data across them either explicitly or using Routers with a load-balancing message distribution policy (more on that in a subsequent post).
MaxQueueLength – This numeric value indicates the maximum queue length. The maximum and default value is 2^31, the minimum value is 1. We’re not enforcing the exact queue length in the March 2009 CTP, but your code should assume that we do. The queue length value is the basis for the calculation of the MaxQueueCapacity, with MaxQueueCapacity = min(MaxQueueLength * MaxMessageSize, 2MB) enforcing the hard 2MB limit that is currently in effect. You don’t need to touch this value unless you’d really want a Queue that’s even more size-constrained.
Overflow – The overflow policy setting becomes relevant when the queue is at capacity and the EnqueueTimeout has expired. It tells the Queue what to do with the message that just came in and that can’t be put into the queue because it’s full. Permitted values:
- RejectIncomingMessage - (default) The message will be rejected and an error status code or SOAP fault will be sent to the sender.
- DiscardIncomingMessage – The sender will get an indication that the message has been accepted, but the message will be dropped on the floor.
- DiscardExistingMessage – The Queue will remove and discard messages from the head of the Queue until the new, incoming message fits in the Queue.
PoisonMessageDrop – [this setting not supported in the March 2009 CTP] This value is a WS-Addressing 1.0 EndpointReference referring to an endpoint that any poison messages will be sent to once the MaxDequeueRetries limit has been exceeded. The EndpointReference may point to any SOAP or plain HTTP endpoint that can accept ‘any’ message.

Phew! Lots of options. The good thing is that most apps should be ok with the defaults.

I know that the 2MB capacity limit is somewhat disappointing and I’m certainly not happy with it. There’s a particular behavior (with bug number) that may occur under very rare circumstances in the underlying replication system, which caused us to play it very safe instead of risking data loss. I don’t think the limit is a showstopper for apps that send notifications and events around – it is a showstopper for apps that want to exchange larger payloads and we’re working to relax that limit and make Queues much, much larger as soon as we can. You can obviously always spool larger data into SDS or one of the Azure storage systems and then send a reference to that data as a message, but it’d be strange for a messaging system to make that a required pattern for data of all sizes. If we’re talking hundreds of megabytes it makes sense, though.

With the due apology out of the way, let’s look at how a policy may be applied to a namespace name – or in other words, how a queue is created in the simplest case (this must be done via HTTPS). The model here is that the client proposes a policy and Service Bus is at liberty to adjust the policy.

If you are intimately familiar with Atom, you’ll notice that the <QueuePolicy> is an extension and isn’t carried as <content>. That’s by intent. <content> is for people, extensions are for apps. We’ll start using <content> in a later milestone, so consider that being reserved.

HTTP/1.1 POST /myapp/q1
Host: clemensv.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect”>
      <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
   </QueuePolicy>
</entry>

If the queue can be created, the response is a 200 and you get the policy back along with any adjustments that the service may make. This is called the “effective” policy, i.e. that’s what the server is using. You also learn about where you can modify the policy, since – as I explained before – the endpoint where you POST the policy to is morphing into the Queue’s tail.

HTTP/1.1 200 OK
Location: https://project-name.servicebus.windows.net/myapp/q1!(queue)
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

If you want to renew the Queue (extend the expiration), take the effective policy, adjust ExpirationInstant and do a PUT to the “self” location.

HTTP/1.1 PUT /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=” http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-04T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

[Bug note: The PUT will return the new effective policy just like the POST response shown above, but the returned Atom <links> aren’t correctly formed. Keep the ones returned from the POST]

If you want to delete the queue, just nuke the policy:

HTTP/1.1 DELETE /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Length: 0

So that’s the policy story for Queues. In the next posts I’ll discuss the REST Queue protocol and the SOAP Queue protocol for how you send message to the queue and get messages out. REST I’ll explain in protocol terms, the SOAP model in .NET programming model terms.

April Fool’s Prediction 2009

Thu, 02 Apr 2009 04:08:14 GMT

My blog has a long memory and therefore I’ll log a prediction for 2014. April Fool’s day is as good as New Year’s for that kind of thing in my book.

I believe that the web browser as “a rectangular window/frame/tab with some content in it” model, along with HTML, will be done in 2014.

I believe that four key, transformational things are underway in the UI space – supported by advances in hardware and distributed computing capacity and distributed storage capacity and network bandwidth - that will kill off the browser paradigm as we know it:

There will be an easy and standard scripting model for real time manipulation / merging / transformation / editing / tagging / recognizing of video sources and the objects in them. Video will replace static pictures.
Autostereoscopic screens will make it into mainstream gear like notebooks and even smartphones.
Touch and proximity sensor based interaction will become the universal form-factor-crossing interaction model for mostly everything.
Graphic display surfaces will increasingly break out of the LCD-panel jail and we'll see use of interactive micro-projection/motion-recognition surfaces everywhere.

I believe that UI will be generated / mashed-up / overlaid / processed entirely and in a personally customizable fashion very close to the user and that any popular user experience shell will concurrently leverage data from at least two dozen different sources.

I believe that interactive local client apps / gadgets / widgets will become the prime battleground for user eyeballs and for advertising display real-estate.

I believe that full-duplex, high-bandwidth, peer-to-peer network communication will be the predominant model for social networking and for sharing data the any of us produces and owns.

Call me crazy.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 1: Fundamentals

Thu, 02 Apr 2009 01:25:18 GMT

In the March 2009 CTP of the .NET Service Bus we’ve added two significant new capabilities, Routers and Queues, that also signal a change of how we’ve thinking about the Service Bus namespace, its capabilities and the road ahead. Before the M5 release, the Service Bus’ primary capability was to act as a Relay between two parties. It’ll absolutely continue to play that role and we’re working to improve that capability further.

The significant shift we’ve made with M5 is that we’ve now started to add long-lived, system-inherent messaging primitives that exist and operate completely independent of any active listener that sits somewhere on some machine and is plugged into the Service Bus. That means that you can now leverage the Service Bus as an intermediary for push-pull translation, or as a publish/subscribe message distribution facility to optimize or facilitate messaging between places that are already “on the Web” or you can set up message distribution scenarios where some messaging destinations are existing Web Services and some receivers require the Service Bus’ relay capability to be reachable.

Before I go into further detail on that, let me explain some of the more philosophical aspects of the model behind Routers and Queues and especially how it relates to the Service Bus namespace that I’ve already discussed to some degree in this post.

Names and Policies

The relationship between any messaging primitive and the Service Bus namespace is established by picking a name in your project’s Service Bus hierarchy - say https://clemensv.servicebus.windows.net/myapp/q1 – and then assign a role to that name. From an astronaut’s perspective, all names in a Service Bus namespace that can theoretically exist do already exist and their role is ‘none’. So when I’m assigning a role to a name, I don’t create the name itself. The name is already there, it’s just in hiding.

That mind-trick is necessary, because we don’t want to burden anyone with creating intermediary names leading up to a name in the hierarchy. In the example I’m using here I would have to first create ‘myapp’ and then create ‘q1’ if we wouldn’t be operating under the assumption that all names you could ever interact with were already existing.

Assigning a role is commonly done using the Atom Publishing Protocol (there’s also a WS-Transfer head that we use for the .NET SDK bits) whereby the POSTed entry contains some form of policy that holds information about what role the name should take on, and what the applicable constraints or operational parameters are. The POST request is sent to the exact URI projection of the name you picked.

Why is that a POST and not a PUT when you already know the URI?

Because once you post a policy to a name, there’s a metamorphosis happening (think “magic little puff of smoke”, not Kafka) that transforms the name into an active messaging primitive. On success, the POST request will yield a 201 response code along with a Location header that indicates the place where you’ll further interact with the policy you just posted. The URI itself is taken over by the primitive.

The picture on the right shows what happens in the case of a queue. As the policy is applied, the queue’s “tail” takes over the URI and two subordinate URIs are created, whereby one serves to interact with the policy and the other one to dequeue messages from the queue’s “head”.

Any name can play any role that’s supported by the system. We currently have a “metadata” role where you just stick an external reference such as a URI or a WS-Address endpoint reference into the name in the registry. We have a “connection point” role that’s established by the WCF listeners as they take over a name to listen on the Service Bus. And we’ve got these two new roles “queue” and “router” that I’m going to explain here.

Queues and Routers

A Router is a publish/subscribe message distribution primitive that allows “push” subscribers to subscribe and get messages that flow into the Router. A Queue is a – well – a queue that accepts messages and holds them until (a) consumer(s) come by and “pull” the messages off the queue. We’re explicitly allowing for Routers to subscribe to Routers and for Queues to subscribe into Routers. The resulting composite is typically quite a bit more powerful than any of the primitives alone. So we call the these capabilities “primitives”, because they explicitly allow for composition.

In the picture on the left you see one possible composition pattern.

We’ve got a number of processing services that we want to load balance jobs across. We also have an auditing service that ought to see and log every single “raw” job message that goes into the system.

The audit service is particularly interested in not losing any messages until they are secured on disk, while the processing services want to get their work pushed to them and run as fast as they can.

The setup is that we’re creating a Router with a message distribution policy of “All” that sends each message to all subscribers. Then we create a secondary Router with a distribution policy of “One”, which sends any incoming message to exactly one randomly selected current subscriber – which solves the load balancing problem for the Processing Service.

For auditing, we also create a Queue that subscribes into the top-level Router that gets all messages and holds them for the Audit Service to pick them up.

The Audit Service would use the peek/lock pattern to get the messages off the queue. That means that the consumer puts an exclusive lock on the message that’s being retrieved and that the message is removed from the view of any competing consumers. The message isn’t gone, though. If the consumer doesn’t acknowledge the message within a minute, the message pops back into view. That means that if the Audit Services were to gets a message but would fumbles it or can’t get it on disk, the message wouldn’t be lost, even in the case of a catastrophic failure. Once the Audit Service can get the message on disk, it deletes the lock and that finally removes the message from the Queue.

So that’s the background on the relationship of Names and Policies and Queues and Routers and how they are designed for composition. In the next posts I’ll go into detail on what the policies for Queues and Routers look like, how you apply them via the SDK programming model or via plain HTTP and how you submit messages into and get messages out of a Router, a Queue or a composite like the one shown here.

[Intermission] Auf der Mauer, auf der Lauer sitzt 'ne kleine Wa! - or: When REST isn't REST - or: Why and How I Care About Standards-Compliance

Wed, 01 Apr 2009 17:40:41 GMT

… seht Euch mal die Wa an, wie die Wa ta kann. Auf der Mauer auf der Lauer sitzt ‘ne kleine Wa!.

It’s a German children’s song. The song starts out with “… sitzt ‘ne kleine Wanze” (bedbug) and with each verse you leave off a letter: Wanz, Wan, Wa, W, – silence.

I’ll do the same here, but not with a bedbug:

Let’s sing:

<soap:Envelope xmlns:soap=”” xmlns:wsaddr=”” xmlns:wsrm=”” xmlns:wsu=”” xmlns:app=””>
   <soap:Header>
         <addr:Action>http://tempuri.org/1.0/Status.set</addr:Action>
         <wsrm:Sequence>
              <wsrm:Identifier>urn:session-id</wsrm:Identifier>
              <wsrm:MessageNumber>5</wsrm:MessageNumber>
          </wsrm:Sequence>
          <wsse:Security xmlns:wsse=”…”>
               <wsse:BinarySecurityToken ValueType=" http://tempuri.org#CustomToken "
                                         EncodingType="...#Base64Binary" wsu:Id=" MyID ">
                          FHUIORv...
                </wsse:BinarySecurityToken>
               <ds:Signature>
                  <ds:SignedInfo>
                      <ds:CanonicalizationMethod Algorithm=" http://www.w3.org/2001/10/xml-exc-c14n#"/>
                      <ds:SignatureMethod Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/ >
                      <ds:Reference URI="#MsgBody">
                            <ds:DigestMethod Algorithm=" http://www.w3.org/2000/09/xmldsig#md5"/>
                            <ds:DigestValue>LyLsF0Pi4wPU...</ds:DigestValue>
                      </ds:Reference>
                 </ds:SignedInfo>
                 <ds:SignatureValue>DJbchm5gK...</ds:SignatureValue>
                 <ds:KeyInfo>
                  <wsse:SecurityTokenReference>
                    <wsse:Reference URI="#MyID"/>
                   </wsse:SecurityTokenReference>
               </ds:KeyInfo>
             </ds:Signature>
         </wsse:Security>
         <app:ResponseFormat>Xml</app:ResponseFormat>
         <app:Key wsu:Id=”AppKey”>27729912882….</app:Key>
    <soap:Header>
    <soap:Body wsu:Id=”MyId”>
          <app:status>Hello, I’m good</app:status>
     </soap:Body>
</soap:Envelope>

Not a very pretty song, I’ll admit. Let’s drop a some stuff. Let’s assume that we don’t need to tell the other party that we’re looking to give it an MD5 signature, but let’s say that’s implied and so were the canonicalization algorithm. Let’s also assume that the other side already knows the security token and the key. Since we only have a single signature digest here and yield a single signature we can just collapse to the signature value. Heck, you may not even know about what that all means. Verse 2:

Better. Now let’s strip all these extra XML namespace decorations since there aren’t any name collisions as far as I can see. We’ll also collapse the rest of the security elements into one element since there’s no need for three levels of nesting with a single signature. Verse 3:

<Envelope>
   <Header>
         <Action>http://tempuri.org/1.0/Status.set</Action>
         <Sequence>
              <Identifier>urn:session-id</Identifier>
              <MessageNumber>5</MessageNumber>
          </Sequence>
          <SignatureValue>DJbchm5gK...</SignatureValue>
          <ResponseFormat>Xml</ResponseFormat>
          <Key>27729912882….</Key>
    <Header>
    <Body>
       <status>Hello, I’m good</status>
     </Body>
</Envelope>

Much better. The whole angle-bracket stuff and the nesting seems semi-gratuitous and repetitive here, too. Let’s make that a bit simpler. Verse 4:

         Action=http://tempuri.org/1.0/Status.set
         Sequence-Identifier=urn:session-id
         Sequence-MessageNumber=5
         SignatureValue=DJbchm5gK...
         ResponseFormat=Xml
         Key=27729912882….
         status=Hello, I’m good

Much, much better. Now let’s get rid of that weird URI up there and split up the action and the version info, make some of these keys are little more terse and turn that into a format that’s easily transmittable over HTTP. By what we have here application/www-form-urlencoded would probably be best. Verse 5:

         method=Status.set
         &v=1.0
         &session_key=929872172..
         &call_id=5
         &sig=DJbchm5gK...
         &format=Xml
         &api_key=27729912882….
         &status=Hello,%20I’m%20good

Oops. Facebook’s Status.set API. How did that happen? I thought that was REST?

Now play the song backwards. The “new thing” is largely analogous to where we started before the WS* Web Services stack and its CORBA/DCE/DCOM predecessors came around and there are, believe it or not, good reasons for having of that additional “overhead”. A common way to frame message content and the related control data, a common way to express complex data structures and distinguish between data domains, a common way to deal with addressing in multi-hop or store-and-forward messaging scenarios, an agreed notion of sessions and message sequencing, a solid mechanism for protecting the integrity of messages and parts of messages. This isn’t all just stupid.

It’s well worth discussing whether messages need to be expressed as XML 1.0 text on the wire at all times. I don’t think they need to and there are alternatives that aren’t as heavy. JSON is fine and encodings like the .NET Binary Encoding or Fast Infoset are viable alternatives as well. It’s also well worth discussing whether WS-Security and the myriad of related standards that were clearly built by security geniuses for security geniuses really need to be that complicated or whether we could all live with a handful of simple profiles and just cut out 80% of the options and knobs and parameters in that land.

I find it very sad that the discussion isn’t happening. Instead, people use the “REST” moniker as the escape hatch to conveniently ignore any existing open standard for tunnel-through-HTTP messaging and completely avoid the discussion.

It’s not only sad, it’s actually a bit frustrating. As one of the people responsible for the protocol surface of the .NET Service Bus, I am absolutely not at liberty to ignore what exists in the standards space. And this isn’t a mandate handed down to me, but something I do because I believe it’s the right thing to live with the constraints of the standards frameworks that exist.

When we’re sitting down and talk about a REST API, were designing a set of resources – which may result in splitting a thing like a queue into two resources, head and tail - and then we put RFC2616 on the table and try to be very precise in picking the appropriate predefined HTTP method for a given semantic and how the HTTP 2xx, 3xx, 4xx, 5xx status codes map to success and error conditions. We’re also trying to avoid inventing new ways to express things for which standards exists. There’s a standard for how to express and manage lists with ATOM and APP and hence we use that as a foundation. We use the designed extension points to add data to those lists whenever necessary.

When we’re designing a ~~RPC~~ SOAP API, we’re intentionally trying to avoid inventing new protocol surface and will try to leverage as much from the existing and standardized stack as we possibly can – at a minimum we’ll stick with established patterns such as the Create/GetInfo/Renew/Delete patterns for endpoint factories with renewal (which is used in several standards). I’ll add that we are – ironically - a bit backlogged on the protocol documentation for our SOAP endpoints and have more info on the REST endpoint in the latest SDK, but we’ll make that up in the near future.

So - can I build “REST” (mind the quotes) protocols that are as reduced as Facebook, Twitter, Flickr, etc? Absolutely. There wouldn’t be much new work. It’s just a matter of how we put messages on and pluck message off the wire. It’s really mostly a matter of formatting and we have a lot of the necessary building blocks in the shipping WCF bits today. I would just omit a bunch of decoration as things go out and make a bunch of assumptions on things that come in.

I just have a sense that I’d be hung upside down from a tree by the press and the blogging, twittering, facebooking community if I, as someone at Microsoft, wouldn’t follow the existing open and agreed standards or at least use protocols that we’ve published under the OSP and instead just started to do my own interpretative dance - even if that looked strikingly similar to what the folks down in the Valley are doing. At the very least, someone would call it a rip-off.

What do you think? What should I/we do?

.NET Services March 2009 CTP - The Service Bus Namespace

Tue, 31 Mar 2009 23:46:35 GMT

With the March CTP, the .NET Service Bus namespace root for each project is taking on a new form that we had already pre-announced into the PDC’08 documentation and that I’ve talked about at PDC and other occasions. Any project’s root URI is now, depending on the chosen transport option, of the form

sb://project-name.servicebus.windows.net/ or
http://project-name.servicebus.windows.net/ or
https://project-name.servicebus.windows.net/

The previous root URI for a project was sb:|http:|https://servicebus.windows.net/services/project-name/, which was clearly sub-optimal for a number of reasons. Some technical, some more philosophical. Technical reasons first:

From the customer perspective, the old structure didn’t allow any project to host “magic” well-known files at the root of the project’s domain. There are unfortunately some protocols that require this, even though it’s broadly considered to be bad practice to bake particular assumptions of the location of resources into protocols. Amongst the “offenders” are robots.txt, favicon.ico and w3c/p3p.xml, but also Adobe Flash’s cross-domain policy crossdomain.xml and – taking a page out of Adobe’s playbook in that case – Microsoft’s Silverlight with ClientAccessPolicy.xml. Therefore the Service Bus was inherently Flash and Silverlight incompatible, unless you served up the apps from within the Service Bus itself. I’m quite sure that there are numerous proprietary protocols used with all sorts of applications that follow a similar pattern and so far those applications could not be exposed through the .NET Service Bus, because there wasn’t any way to put anything at the root. Now there is, even though it requires a little WCF trick that I’ll explain in a separate post specifically addressing crossdomain.xml and ClientAccessPolicy.xml.
The most important reason for us was that we need to be able to scale out the system into an “infinite” number of partitions (or “scale-units”) inside and across data-centers. Therefore, each project now gets its very own DNS record pointing to the scale-unit(s) where the project is hosted. The DNS system we’re leveraging is the one that’s used across most Windows Live properties and has a number of very interesting characteristics that will allow us – over time - to optimize the Service Bus traffic flow and to drive down end-to-end latency.

What matters even more to us from an “aesthetics” and design perspective is that we really want a project’s namespace to be clean, isolated, and want to avoid any artificial, system imposed pollution in the namespace. The fact that there are sound technical reasons supporting that desire is even more helpful to create consensus around this.

At its core, the Service Bus namespace is a federated, hierarchical service registry, whose structure is dictated and owned by “the project”. The difference between the Service Bus namespace and a “classic” service registry system like DNS or UDDI or LDAP is that services or messaging primitives are (usually) not only referenced by the registry, but they are projected straight into the registry, so that you can interact with the registry and those services or messaging primitives projected into the registry using similar or identical programming interfaces and within the scope of a unified namespace. We intentionally blur the distinction.

The service registry’s “discovery” (or “browsing”) protocol is based on a simple, conceptually infinite, nested hierarchy of ATOM 1.0 feeds. (In case you are wondering: Yes, we’ve got work lined up to provide flattened, non-hierarchical discovery overlaid over the hierarchy.) If you project an ATOM 1.0 endpoint into any name in the name hierarchy and choose to make that endpoint discoverable, the transition from the discovery hierarchy across the Service Bus namespace into the discovery of the resource hierarchy provided by your endpoint is absolutely seamless. That’s a design point.

What makes the namespace “federated” is that services or messaging primitives can be projected into the shared namespace from “anywhere”. Typically, the path portion of a URI represents a set of relatively tightly collocated set of resources that are residing across a web farm or a database cluster with the authority portion identifying (directly or indirectly) the target cluster.

.NET Service Bus URIs obviously look exactly like that, but they are quite different.

Imagine you have a distributed setup with 3 different “order processing” systems: One for the U.S. near Seattle, one for the EU near Düsseldorf, and one for the SEA region in Singapore. Let’s also assume that I’m not talking about a multi-national mega-corporation here, but about a trading company with some 40 people that happens to have these three offices. Let’s also assume that they are using a customized ISV application that has been adapted for them by a (plug!) Microsoft Certified Partner and that application is primarily designed to run on local servers. Let’s also assume that it would be difficult (or prohibitively expensive) for them to get a static IP address and a corresponding, secure network setup that would allow them to wire up the application at the respective sites to the outside world. If you are helping customers with business apps, you might find that scenario to be resonating with your experience.

The .NET Service Bus can help with the connectivity issues by allowing to project the endpoints into the Service Bus namespace. That means that the application’s endpoints are listening for messages on and inside the .NET Service Bus instead of some local network listener on-site. There is no need to open any inbound firewall port and no need to do anything to the NAT setup and no need to do anything with DNS. Clients talk to those endpoints. The Service Bus namespace helps with getting those applications organized in a way that you can look at the resulting distributed system as “one” even though it spans sites:

http://littletradingcorp.servicebus.windows.net/orders/seattle/
http://littletradingcorp.servicebus.windows.net/orders/dusseldorf/
http://littletradingcorp.servicebus.windows.net/orders/singapore/

In combination with the .NET Access Control service, you can now overlay a single set of access control rules over the base scope http://littletradingcorp.servicebus.windows.net/orders/which yields a centrally manageable access control overlay over these three services, even though the actual servers and endpoints are spread once around the world.

What makes the naming system very different from DNS is that the .NET Service Bus naming system names endpoints and not hosts. Let’s say that each site also hosts a local “human resources” software; at that company size that may very well be an application that runs on the respective branch manager’s desktop machine or on a small server. That system is quite naturally distinct from from the order processing system and its reasonable safe to assume that the company wouldn’t want to collocate that system with the order processing system. Let’s project these into the namespace as well and we’ll certainly assume they have different Access Control rules that apply to the respective root scope:

http://littletradingcorp.servicebus.windows.net/hr/seattle/
http://littletradingcorp.servicebus.windows.net/hr/dusseldorf/
http://littletradingcorp.servicebus.windows.net/hr/singapore/

If we were trying to provide direct access to the “orders” and “hr” endpoints using HTTP with a “normal” setup, we would either have to have – especially with HTTPS – two public, static IP addresses for each site that are mapped to the respective machines or we’d have to use some gateway server that would dispatch the requests locally based on a host-header (requiring distinct DNS entries) or on the path prefix or we’d have to resort to non-standard ports – and we’d open up the floodgates for arbitrary and potentially malicious inbound traffic on the opened ports. And if we had that we would have to map these IP addresses into some DNS naming structure. That’s quite a bit of networking work. Not necessarily complicated for someone who is well versed in these matters, but at the very least it’s a significant cost point. The resulting complication grows with each endpoint, since DNS identifies the public IP gateway and not the endpoint. So things get trickier and trickier. If we want to help small and medium businesses to “go digital” and intercommunicate more efficiently over the web (Fax is still the king in very many places and businesses), all those networking acrobatics don’t scale well.

Mind that I argue that not all business owners are happily accepting the notion of putting all their data and apps into someone else’s data center or "into the cloud”. Mind also that I don’t think I’m contradicting myself here. The .NET Service Bus is the “+” in what Microsoft calls “Software+Services”. It facilitates communication between places of which either can be in the cloud or at a customer-controlled site, it’s not the place where you “put data”.

So much for this post – in the next we’re going to look at the new March 2009 CTP Routers and how they interact and integrate with the namespace structure.

.NET Services March 2009 CTP - "Milestone 5"

Tue, 31 Mar 2009 23:45:13 GMT

Our deployment team reports that they’re done with the last touches on our new release, that the servers are happily humming with the new bits, and that the new SDK is posted.

The .NET Services Developer Center has been updated as well. You can get the SDK bits from here [1]. We also have a runtime-only redistributable package that sits here [2].

What changed? Quite a bit. I’ll be updating you about the concrete changes in the .NET Service Bus beyond what we’ve got in the docs here on my blog over the next few days and also go into detail on some of the design considerations for the newly added functionality and the service APIs, specifically also on the significantly expanded REST capabilities.

People who’ve been following .NET Services or even BizTalk Services for a while will immediately notice that we’ve introduced a number of breaking changes in this release and that your existing applications will require some changes. The most important change we’ve made across the services and .NET Service Bus in particular is that we’re now giving every project a clean “root” URI:

Before the March CTP, the “root” of your namespace resided at: http://servicebus.windows.net/services/project-name/
With the March CTP, we’ve moved the project-name into the DNS name and drop the ‘services’ segment: http://project-name.servicebus.windows.net/

With that, we’re giving you complete control over your Service Bus namespace and we’re making every effort to leave you in the driver’s seat on the namespace design and to avoid hijacking URIs for “magic” functionality. What that means – and I will elaborate on that topic be showing you some code in the next few days – is that you can now do things like putting virtual robots.txt or Silverlight ClientAccessPolicy.xml or Flash crossdomain.xml endpoints at the root, enabling cross-domain scripting against relayed HTTP services.

The good news is that we’ve made no visible changes to the WCF bindings for the .NET Service Bus that we talked about at PDC. We’ve improved some of them in significant ways under the hood, but we didn’t touch the API surface in any way you ought to notice in your apps. A very important set of bug fixes that went in in that area is that we’re now respecting all size quotas and timeouts you set on the bindings. We didn’t really do a good job there in previous releases. When you ask for a 10 second “open” timeout for a binding that’ll actually be effective now. Before you were stuck with a minute no matter what you set the values to.

The “flagship” WCF binding of the .NET Service Bus, the NetTcpRelayBinding got the most attention. The binding knows how to do a particularly awesome networking stunt when you set the ConnectionMode=TcpConnectionMode.Hybrid. It will attempt (and mostly succeed) snapping a direct socket between two parties even through both parties may be hidden behind opposing firewalls and NATs. As we’ve been analyzing some bugs in this latest milestone we’ve found that the established connections were only working well for one-way communication, but not for full-duplex or even request-reply conversations. We’ve fixed that. What we’ve also added is a status indicator by which you can tell whether the connection is in “relayed” mode or whether it has been upgraded to “direct” mode. The new mechanism is shown in the Direct Connect sample, but I’ll quote that here for you as I go through the features in more detail.

A commonly seen customer issue is that quite a few people don’t have any leverage to get the outbound TCP ports that we require for outbound communication opened on an upstream firewall (remember: no inbound open firewall ports required; you can stay stealthy). We’ve started to address this across all our bindings so that the .NET Service Bus client and listener endpoints are a bit more aggressive in trying to get out to the .NET Service Bus. That particular feature is in the bits, is enabled by default and we expect (hope) that people who are trying to connect from within tightly managed network environments with outbound port restrictions will see things “just work”. We’re eager to find out whether people who’ve been reporting DNS resolution errors for servicebus.windows.net or nothing but obscure 1-minute timeout failures are finding that things have improved for them – albeit connections may not be as snappy as without such port restrictions in place.

Absolutely new in this release are a set of composable communication primitives that I’ve talked about and demoed at MIX: Queues and Routers. There’s a whole lot of philosophy behind the way those primitives are designed, what those primitives indicate about where we’re going with the .NET Service Bus, how they integrate into the namespace, and there’s a lot to say on how you can interact with them and compose them and therefore I’ll talk about the namespace as well as Queues and Routers in detail and in a few separate posts starting later today.

[1] http://www.microsoft.com/downloads/details.aspx?FamilyID=8d1d1d5e-1332-4186-b33f-26d053759e49&displaylang=en
[2] http://www.microsoft.com/downloads/details.aspx?FamilyID=4eff38b3-fca0-4940-a014-461631152e5d&displaylang=en

17 Months

Thu, 06 Nov 2008 00:34:09 GMT

Now that I’m blogging again I thought it’d be time for a little update.

The Service Bus Bindings and WAS/IIS

Wed, 05 Nov 2008 23:16:43 GMT

We’ve been getting some questions along the lines of “I am hosting a service as xyz.svc in IIS and have changed the config to use on of the Service Bus bindings, but the service never gets called?”

That’s right. It doesn’t. The reason for that is that we don’t yet have WAS/IIS integration for any of the Service Bus bindings in the November 2008 CTP. Enabling the WCF WAS activation scenario that puts the NetTcpRelayBinding and friends on par with their WCF siblings is on our work backlog for the next major milestone.

It’s worth considering for a moment what that integration requires. Fundamentally, all of the Relay bindings replace the local TCP or HTTP listener with a listener that sits up in the cloud and services then connect up to that listener to create an inbound route for received messages. That’s similar to how local services interact with WCF’s shared TCP listener or HTTP.SYS, but there are quite a few important differences. First, all Relay listeners need to acquire and present an Access Control token when they start listening on the Service Bus. In contrast, the local listener facilities are ACL’d using the local or domain account system and use the Windows process identity to decide on whether a process may or may not listen on a particular port and/or namespace. Second, since the actual listener is off-machine, we need to spin up the connection as the IIS/WAS host spins up and need to make sure that the connection is kept alive and aggressively reconnects when dropped for any reason. That’s something you don’t really have to worry much about when the listener sits right there on the same machine as your own service and the connection is a named pipe. Third, the local listeners listen on a particular host address and port; the Relay listeners listen on a leaf of a namespace tree and that namespace may be shared amongst many listeners living on a multitude of different machines in different locations. Fourth, ... well you get the picture.

Bottom line: Not having support for WAS activation and xyz.svc service endpoints is by no means an oversight. It’s on the list.

.NET Services: MSDN Developer Center

Wed, 05 Nov 2008 19:27:31 GMT

The MSDN Developer Center for .NET Services is the first stop to go to for technical information on the Service Bus, the Access Control Service and the Workflow Service.

There quite a bit of documentation for “my” feature area, the .NET Service Bus, including description of all the bindings and most of the object model surface area. Since we had quite a bit of object model churn up until a few weeks before PDC as we’ve exploded the former, singular RelayBinding into two handful of WCF-aligned bindings, the reference documentation isn’t yet in the familiar MSDN reference format and also doesn’t yet work with Visual Studio’s “F1”. We’re obviously going to address that in the next major milestone now that the dust is settling a bit and the programming model is already quite a bit closer to what we want it to be for our “V1” release.

.NET Services - Two things you may not have noticed yet....

Mon, 03 Nov 2008 05:03:21 GMT

In the sea of PDC 2008 announcements you may have missed the following two signficant developments:

For the past 2 months our team has worked very closely with our partners at Schakra on the Java SDK parts and with ThoughtWorks on the Ruby parts. These are the first baby steps and these two SDKs cover only a small subset of the capabilities of the .NET SDK so far. That's merely a function of when we started with these projects and how far we've gotten with the required protocol support; we want and we will take this a lot further over the next development milestones. In the end, the .NET Services fabric ought not to care much what language the senders and listeners are written in and what platform they run on. We're building a universal services platform. We're taking Java and Ruby very seriously and have a few more platforms on the list for which we want to add explicit support.