Clemens Vasters

TechEd: ASI204 Windows Azure Platform AppFabric Overview

Tue, 01 Jun 2010 21:15:06 GMT

Room 398, Tuesday June 8
3:15pm-4:30pm

Session Type: Breakout Session
Track: Application Server & Infrastructure
Speaker(s): Maggie Myslinska
Level: 200 – Intermediate

Come learn how to use Windows Azure AppFabric (with Service Bus and Access Control) as building block services for Web-based and hosted applications, and how developers can leverage services to create applications in the cloud and connect them with on-premises systems.

If you are planning on seeing Juval’s and my talk ASI304 at TechEd and/or if you need to know more about how Windows Azure AppFabric enables federated cloud/on-premise applications and a range of other scenarios, you should definitely put Maggie’s talk onto your TechEd schedule as well.

TechEd: ASI302 Design Patterns, Practices, and Techniques with the Service Bus in Windows Azure AppFabric

Fri, 28 May 2010 11:40:19 GMT

Room 265, Thursday June 10
9:45AM – 11:00 AM

Session Type: Breakout Session
Track: Application Server & Infrastructure
Speaker(s): Clemens Vasters, Juval Lowy
Level: 300 - Advanced

The availability of the Service Bus in Windows Azure AppFabric is disruptive since it enables new design and deployment patterns that are simply inconceivable without it, opening new horizons for architecture, integration, interoperability, deployment, and productivity. In this unique session organized especially for Tech·Ed, Clemens Vasters and Juval Lowy share their perspective, techniques, helper classes, insight, and expertise in architecting solutions using the service bus. Learn how to manage discrete events, how to achieve structured programming over the Service Bus buffers, what options you have for discovery and even how to mimic WCF discovery, what are the recommended options for transfer security and application authentication, and how to use AppFabric Service Bus for tunneling for diagnostics or logging, to enabling edge devices. The session ends with a glimpse at what is in store for the next versions of the service bus and the future patterns.

Yes, that's Juval and myself on the same stage. That'll be interesting.

NT Konferenca 2010 - Windows Azure Slidedecks

Tue, 25 May 2010 07:08:59 GMT

I put the slides for my talks at NT Konferenca 2010 on SkyDrive. The major difference from my APAC slides is that I had to put compute and storage into one deck due to the conference schedule, but instead of purely consolidating and cutting down the slide count, I also incorporated some common patterns coming out from debates in Asia and added slides on predictable and dynamic scaling as well as on multitenancy. Sadly, I need to rush through all that in 45 minutes today.

Windows Azure Speaking Tour Slides

Sun, 16 May 2010 08:31:17 GMT

I'm on a tour through several countries right now and I'm talking to ISVs about the Windows Azure platform, its capabilities and the many opportunities ISVs have to transform the way they do business by moving to the cloud. The first day of the events is an introduction to the platform at the capability level; it's not a coding class, that would be impossible to fit.

I've shared the slides on SkyDrive. Steal liberally if you find the material useful.

Hosting Service Bus endpoints in IIS and Server AppFabric (Dublin)

Sun, 09 May 2010 23:45:33 GMT

My office neighbor, our Service Bus Test Lead Vishal Chowdhary put together a bundle of code and documentation for how to use Service Bus with Server AppFabric and IIS 7.5. Here: http://code.msdn.microsoft.com/ServiceBusDublinIIS

UA 875 SEA NRT

Sun, 09 May 2010 07:28:04 GMT

It’s not a revival of my career as a traveling talking head, but it sure feels a bit like old days. I’m on United 875 from Seattle to Tokyo/Narita right now, somewhere above the North Pacific. A few hours more to go and then I’ll be connecting to Singapore and I’m guessing that I’ll be at my hotel sometime between 1am and 2am on Monday morning.

I’ll spend two days in Singapore as part of a Windows Azure ISV workshop series that has been organized by our field colleagues in the APAC region; the first day I’ll be presenting the all-up Windows Azure Platform –Compute, Storage, Management and Diagnostics, Database, Service Bus, Access Control, and the additional capabilities we’ll be adding over the next several months. On day two, I’ll be meeting for 1:1s with a range of customers about their plans to move applications to the cloud. That pattern will repeat over the next two weeks in Kuala Lumpur/Malaysia (this Thu/Fri), in Manila/Philippines (next Mon/Tue), and in Seoul/South Korea (next Wed/Thu). From Seoul onwards, some of my colleagues will take over and go to Sydney and Auckland, while I’m flying further westwards to Europe to speak at the NT Konferenca in Slovenia before returning to Seattle after a short stopover in Germany to see the folks.

Once I’m back in Seattle I’ve got 5 days at the office to debrief and prep for TechEd North America and then it’s off to New Orleans for the week and then, after a weekend stopover in Seattle, I’m off to the NDC 2010 conference in Oslo/Norway. It’s definitely the most flying I’ve done since I work for Microsoft.

The 1:1 meeting opportunities at the workshops in Asia, at NT Konferenca, at TechEd, and at the NDC2010 is what this tour is all about for me. It’s about reaching out and feeling the pulse of the customer landscape. That is very, very different from back when I was a traveling talking head explaining the platform. That’s not to say that there wasn’t a lot of value in teaching back then. Conference and workshop attendees learned a lot and Microsoft got the word out.

Back in the day it was all about outbound communication; and you can probably tell by my blog having been practically dead over the last 2-3 years that that hasn’t been my focus anymore. But what about the other way? You’d think that Microsoft is overwhelmed with incoming data and swims in requirements and scenarios and customer input. That’s right, we are. As a company. The problem for us in the product teams is that we’re getting much of that data in a very indirect and filtered form, which is simply because we are a very big and global company. Another problem (or danger) is to be tricked into believing that the entire truth can be found online and via feedback mechanisms like forums or even Facebook and Twitter. None of that replaces an hour of high-bandwidth, eye-to-eye conversation with someone whose business is not primarily (or not at all) about software, but for whom software is a mere necessity to get their products and services to their customers. It’s my firm belief that you have to get into a plane sometimes and go where the customers are. Luckily, my boss agrees. So here I am.

Some of the customers I’ll be talking to in 1:1s already have firm plans and want to talk architecture, some don’t know whether it’s a good decision for them and want to dig into details, some have made a decision for the cloud but went with another competing platform, and some don’t want or can’t move anything to our cloud yet, because that platform isn’t doing what they need. The latter two types of customers are the ones I’m most looking forward to meet.

The events in Asia are run in relatively small venue and my understanding is that my 1:1 days are booked out. I don’t think that’s true for the events in Slovenia, New Orleans, and in Oslo. If you plan on going and you are interested in spending an hour talking about what you’d like to see in Windows Azure and specifically in Windows Azure AppFabric then shoot me an email to clemensv at microsoft dot com in the next few days. And if you’re not going tell your friends who do ;-)

(Oh, and: United, did you buy your business class seats third-hand from TWA after they picked them up from PanAm?)

United EmPower and my ThinkPad T60p: iGo everywhereMAX

Sun, 09 May 2010 07:26:32 GMT

When I thought about what to pack for this trip on Friday night, I checked once again what aircraft I’d be on: United 777 International configuration. Oh, the horror! “EmPower”. In case you don’t know what that means: EmPower is a proprietary power outlet providing 15V DC with a maximum of 75W. It practically only exists on aircraft and it’s a gold-mine for companies selling extra power-supplies for Notebooks and practically any other electronic device that you might use while on an aircraft – and lucky enough to sit somewhere towards the front of the aircraft.

Now, if it’s not yet obvious, let me be clear: I believe that having to buy some extra adapter to power my notebook in business class is a scandal. It may be my previous Lufthansa bias, but I think these outlets ought to be following some real-world standard, even if that were 115V/60Hz AC. But none of the complaining is helping me power the notebook, so I broke down and bought the required adapter at SeaTac airport this morning: iGo everywhereMAX. 139.99 + tax. One-hundred-and-forty bucks. For a power adapter. That’s a rip-off. This should be illegal.

It does what it is advertised to do. There’s a standard U.S. power cable and a “works on the plane and in your car and on your boat” adapter that’s EmPower compatible once you pull the standard cigarette-lighter adapter off. The package comes with a plastic bag of notebook adapters and the coverage is indeed very good; all the notebooks in the family and the ones I use for work are on the list. Acer, Dell, Toshiba, Sony, Compaq, Gateway, HP, …

I’d be unhappy if I owned a Mac, because there doesn’t seem to be any support for Apple machines. I’m a PC, I have power on this flight :)

You’d think that the extra power outlet for devices (phones, music players, etc) would be based on something reasonably standard – USB maybe. Noooo! It’s some weird only-compatible-with-this-power-adapter plug that iGo must have made up while laughing evilly. You can get a USB plug, but you will have to buy that extra. Also, the AC power cord won’t do me any good in countries that aren’t using the two-blade U.S. power plug (—> most countries). Lucky me, I already brought a universal adapter along, because my new $140-plus-tax electric-engineering masterpiece wouldn’t me of much use on the ground.

Given the number of hour I’ll spend on aircraft in the upcoming weeks I wonder whether my execusphere is going to let me expense it.

Writing talent is a terrible thing to waste. Not that I have any.

Sun, 09 May 2010 07:25:44 GMT

I really should pick up blogging again. Between the ephemeral twitter, foursquare, and facebook grows the need for something more sticky.

The Rough Setup Script for PDC09 SVC18 - Getting DinnerNow! to run on Windows Azure

Mon, 23 Nov 2009 23:42:16 GMT

At the bottom of this post you’ll find the DinnerNow version that I’ve been using for my PDC09 talk. The video of that talk is now available at http://microsoftpdc.com/Sessions/SVC18 and I recommend that you listen to the talk for context.

The DinnerNow drop I’m sharing here is a customized version of the DinnerNow 3.1 version that’s up on CodePlex. If I were you, I’d install the original version and then unpack my zip file alongside of it and then use some kind of diff tool (the Windows SDK’s WinDiff tool is a start) to look at the differences between the versions. That will give you a raw overview of what I had to do. You’ll find that I had to add and move a few things, but that the app didn’t change in any radical way.

Remember that looking at the code is more important that making it run. There’s one particular challenge you’d have right now with the Windows Azure CTP and that’s getting the two (!) Windows Azure compute tokens needed for separating out the web and the service tier as I’ve done here. It’s not difficult to consolidate the Web and the Web Service tier into a single role, but since I had to do the migration within a short period of time, I chose to split them up.

FWIW, I time-boxed the migration to 3 work days – which included learning about what our buddies over in SQL Azure had done in the past months — and that turned out to be a comfortable fit in terms of time.

Another function of time-boxing is that you’re finding me disabling security on most endpoints, including disabling the Access Control integration with Service Bus for most endpoints by setting the relayClientAuthenticationType attribute on the respective binding elements to None.

I know that’s a sin, but I didn’t want to cause too much churn in the first iteration. The original version of DinnerNow is conveniently using Windows authentication/authorization for its communication paths. While that’s ok for a LAN setup, things get more complicated for an actual WAN setup that the DinnerNow scenario calls for. That would spawn a wholly different discussion that shines the spotlight on our Access Control service and why it’s useful – even required – for that scenario. In order not to overwhelm everyone, I left that out for this round and will revisit that aspect in the next weeks – or maybe one of our (aspiring?) MVPs or RDs will beat me to it.

I’m also going to work with the guys who wrote DinnerNow to find a way to host this modified version of Dinner Now with the on-premise runtime bits expressly not on my primary dev machine, where they’d live now.

Here what you need to do to get it to run

I know this is rough. Writing up the long version of this is going to take some time and I prefer getting the bits to you early over me sitting here writing pages of docs. Maybe you can even help ;-)

First, you’ll need to go to the Windows Azure portal and get the SDKs and tokens/accounts. The Getting Started page has all the data and links you need so I’m not going to repeat them here in much detail. You will need at least one Windows Azure compute account (apply here), one SQL Azure account (apply here), and an AppFabric account (no application needed, just log in w/ LiveID).
Download and install the regular version DinnerNow 3.1 from Codeplex. This will drop a “Configure DinnerNow 3.1” shortcut on your desktop. Run that, install all prerequisites and make sure DinnerNow runs locally before you proceed.
You will later need the databases that the setup created in your local SQLEXPRESS instance by setup. You’ll have to make a few changes, though.
1. First, (download, install, and) open SQL Server Management Studio, connect to your SQL Server Express instance and switch to “SQL Server and Windows Authentication mode” on the Server Properties under Security. Then you’ll need to go to to the Security settings and either create a new account and grant it all rights on the aspnetdb database or just enable the ‘sa’ account and set its password.
2. Then you need to find the “SQL Server Configuration Manager” and enable TCP for your SQLEXPRESS instance like this. The default port will be 1433. If you have a full SQL Server instance on your dev machine and it’s configured for TCP the easiest is to suspend that for the moment and allow the SQLEXPRESS instance to squat the port.
Unpack the ZIP file appended below into a directory on your machine. At this point it should be ok to override the existing DinnerNow directory, but I’d keep things side-by-side for reference. If you copy side-by-side, grab the ./solution/DinnerNow – Web/DinnerNow.WebUX/images/’ directory from your local installation and copy it into the location where you unzipped the file here. I left out the images due to their size. And just as with the normal DinnerNow installation you’ll find a solution file named “DinnerNow - Main.sln” in the unpacked directory – open that in Visual Studio 2008 (not 2010!) because you’ll have to make some changes and edits.
If you are lucky enough to have two Windows Azure compute accounts, you can skip this step. Otherwise, you will have to restructure the application a bit:
1. In the “DinnerNow – WA” solution branch where the Windows Azure deployment project reside you’ll have to consolidate the DinnerNow.WindowsAzure project and the DinnerNow.WindowsAzureAppSrv projects into one by replicating the DinnerNow.DBBridge reference into the DinnerNow.WindowsAzure project and abandoning/deleting the rest.
2. In the “DinnerNow – Web” solution branch you will have to modify the DinnerNow.WebUX project by merging the DinnerNow.ServiceHost project from the “DinnerNow -ServicePortfolio2” branch into it, including merging the config files. In the original DinnerNow the hosting default is that the ServiceHost project lives in the ./services subdirectory of the WebUX app. You can also do it that way, but you’ll have to change the respective client URIs to point to the right path.
In the ./database directory is a file called SQLAzureImport.sql. That’s the exported and customized script for the DinnerNow restaurants and menus database. Create a new database (1GB is enough) and load the DB with this script. You can do this with the command line or with SQL Management Studio. The SQL Azure docs will tell you how.
Now you’ll need to do a range of search/replace steps across the whole project. These are mostly in *.config files - a few places are in the code, which I count as bugs, but those are faithfully carried over from the original:
1. Find all occurrences of sqlazure-instance and replace them with your unqualified SQL Azure server name (might look like this: tn0a1b2c3d)
2. Find all occurrences of sqlazure-dbname and replace them with your SQL Azure database name
3. Find all occurrences of sqlazure-acct and replace them with your SQL Azure administrator username
4. Find all occurrences of sqlazure-password and replace them with your SQL Azure administrator password
5. Find all occurrences of appfabricservicebus-ns and replace them with your unqualified AppFabric namespace name
6. Find all occurrences of appfabricservicebus-key and replace them with your AppFabric Service Busissuer key
7. Find all occurrences of windowsazuresvcrole-acct and replace them with the name of your Windows Azure compute account. If you have just one, use that (given you’ve done the rework in step 4), if you have two use the account-name where you will host the service tier.
8. Find all occurrences of sqlserver-password and replace them with your localSQL Server Express instance’s ‘sa’ account password.
Do a full batch Rebuild of the whole project
Go to the “DinnerNow –WA” solution and publish the project(s) to your Windows Azure compute account(s). If you had to consolidate them you’ll have one package to deploy, if you left things as they are you’ll have two packages to deploy. You can also run these packages in the local DevFabric to test things out.
The executables you need to run are going to be dropped into the .\bin directory by the build. You need to run all 6 apps – but you could run them on 6 different machines – the two workflow hosts each assume the local presence of the DinnerNowWF database:
1. CloudTraceRecorder.exe – this is the simple event listener app. You can run this right away to observe the apps starting up inside of Azure as they write events to the event listener. You can and should run this as you deploy. You can run any number of instances of CloudTraceRecorder anywhere.
2. PortBridge.exe – this is the on-premise bridge-head for bridging to your local SQL Server Express instance so that the cloud application can get at its membership database that you host for it on your machine. After the search/replace steps you will notice that you have modified connection strings that point to a SQL Server role peeking out of your *AppSrv role. The secret ingredient is in the DinnerNow.DBBridge role that’s listening for TCP connections on behalf of your on-premise SQL Server and that connects them down to your local server with the logic in Microsoft.Samples.ServiceBus.Connections. This is the same code that’s in PortBridge.
3. DinnerNow.OrderProcessingHost.exe is the (new) host application for the workflow that handles the order process.
4. DinnerNow.RestaurantProcessingHost.exe is the (new) host application for the workflow that handles the restaurant process.
5. DinnerNowKiosk.exe is the only slightly modified version of the DinnerNow in-restaurant kiosk
6. Not in .\bin but rather to be started/deployed from VS is the also just slightly modified Windows Mobile app for the delivery app

Please also mind that the DinnerNow Powershell support and the other test and diagnostics capabilities haven’t been touched here, yet.

Oh, and … this is provided as-is … I’ll do my best to discuss some of the patterns over the next several weeks, but I don’t have time to provide 1:1 support.

Here’s the code:

DinnerNow-SVC18-PDC09.zip (2.35 MB)

Port Bridge

Wed, 18 Nov 2009 17:37:28 GMT

Building “hybrid” cloud applications where parts of an an app lives up in a cloud infrastructure and other parts of the infrastructure live at a hosting site, or a data center, or even in your house ought to be simple – especially in this day and age of Web services. You create a Web service, make it accessible through your firewall and NAT, and the the cloud-hosted app calls it. That’s as easy as it ought to be.

Unfortunately it’s not always that easy. If the server sits behind an Internet connection with dynamically assigned IP addresses, if the upstream ISP is blocking select ports, if it’s not feasible to open up inbound firewall ports, or if you have no influence over the infrastructure whatsoever, reaching an on-premise service from the cloud (or anywhere else) is a difficult thing to do. For these scenarios (and others) our team is building the Windows Azure platform AppFabric Service Bus (friends call us just Service Bus).

Now – the Service Bus and the client bits in the Microsoft.ServiceBus.dll assembly are great if you have services can can be readily hooked up into the Service Bus because they’re built with WCF. For services that aren’t built with WCF, but are at least using HTTP, I’ve previously shown a way to hook them into Service Bus and have also demoed an updated version of that capability at Sun’s Java One. I’ll release an update for those bits tomorrow after my talk at PDC09 – the version currently here on my blog (ironically) doesn’t play well with SOAP and also doesn’t have rewrite capabilities for WSDL. The new version does.

But what if your service isn’t a WCF service or doesn’t speak HTTP? What if it speaks SMTP, SNMP, POP, IMAP, RDP, TDS, SSH, ETC?

Introducing Port Bridge

“Port Bridge” – which is just a descriptive name for this code sample, not an attempt at branding – is a point-to-point tunneling utility to help with these scenarios. Port Bridge consists of two components, the “Port Bridge Service” and the “Port Bridge Agent”. Here’s a picture:

The Agent’s job is to listen for and accept TCP or Named Pipe connections on a configurable port or local pipe name. The Service’s job is to accept for incoming connections from the Agent, establish a duplex channel with the Agent, and pump the data from the Agent to the actual listening service – and vice versa. It’s actually quite simple. In the picture above you see that the Service is configured to connect to a SQL Server listening at the SQL Server default port 1433 and that the Agent – running on a different machine, is listening on port 1433 as well, thus mapping the remote SQL Server onto the Agent machine as if it ran there. You can (and I think of that as to be more common) map the service on the Agent to any port you like – say higher up at 41433.

In order to increase the responsiveness and throughput for protocols that are happy to kill and reestablish connections such as HTTP does, “Port Bridge” is always multiplexing concurrent traffic that’s flowing between two parties on the same logical socket. When using Port Bridge to bridge to a remote HTTP proxy that the Service machine can see, but the Agent machine can’t see (which turns out to be the at-home scenario that this capability emerged from) there are very many and very short-lived connections being tunneled through the channel. Creating a new Service Bus channel for each of these connections is feasible – but not very efficient. Holding on to a connection for an extended period of time and multiplexing traffic over it is also beneficial in the Port Bridge case because it is using the Service Bus Hybrid connection mode by default. With Hybrid, all connections are first established through the Service Bus Relay and then our bits do a little “NAT dance” trying to figure out whether there’s a way to connect both parties with a direct socket – if that works the connection gets upgraded to the most direct connections in-flight. The probing, handshake, and upgrade of the socket may take 2-20 seconds and there’s some degree of luck involved to get that direct socket established on a very busy NAT – and thus we want to maximize the use of that precious socket instead of throwing it away all the time.

That seems familiar?!

You may notice that SocketShifter (built by our friends at AWS in the UK) is quite similar to Port Bridge. Even though the timing of the respective releases may not suggest it, Port Bridge is indeed Socketshifter’s older brother. Because we couldn’t make up our mind on whether to release Port Bridge for a while, I had AWS take a look at the service contract shown below and explained a few principles that I’m also explaining here and they had a first version of Socketshifter running within a few hours. There’s nothing wrong with having two variants of the same thing.

How does it work?

Since I’m publishing this as a sample, I obviously need to spend a little time on the “how”, even I’ll limit that here and will explain that in more detail in a future post. At the heart of the app, the contract that’s used between the Agent and the Service is a simple duplex WCF contract:

    [ServiceContract(Namespace="n:",
Name="idx", CallbackContract=typeof(IDataExchange),
SessionMode=SessionMode.Required)] public
interface IDataExchange { [OperationContract(Action="c",
IsOneWay = true, IsInitiating=true)] void Connect(string i);
[OperationContract(Action = "w",
IsOneWay = true)] void Write(TransferBuffer d);
[OperationContract(Action = "d",
IsOneWay = true, IsTerminating = true)] void Disconnect();
}

There’s a way to establish a session, send data either way, and close the session. The TransferBuffer type is really just a trick to avoid extra buffer copies during serialization for efficiency reasons. But that’s it. The rest of Port Bridge is a set of queue-buffered streams and pumps to make the data packets flow smoothly and to accept inbound sockets/pipes and dispatch them out to the proxied services. What’s noteworthy is that Port Bridge doesn’t use WCF streaming, but sends data in chunks – which allows for much better flow control and enables multiplexing.

Now you might say You are using a WCF ServiceContract? Isn’t that using SOAP and doesn’t that cause ginormous overhead? No, it doesn’t. We’re using the WCF binary encoder in session mode here. That’s about as efficient as you can get it on the wire with serialized data. The per-frame SOAP overhead for net.tcp with the binary encoder in session mode is in the order of 40-50 bytes per message because of dictionary-based metadata compression. The binary encoder also isn’t doing any base64 trickery but treats binary as binary – one byte is one byte. Port Bridge is using a default frame size of 64K (which gets filled up in high-volume streaming cases due to the built-in Nagling support) and so we’re looking at an overhead of far less than 0.1%. That’s not shabby.

How do I use it?

This is a code sample and thus you’ll have to build it using Visual Studio 2008. You’ll find three code projects: PortBridge (the Service), PortBridgeAgent (the Agent), and the Microsoft.Samples.ServiceBus.Connections assembly that contains the bulk of the logic for Port Bridge. It’s mostly straightforward to embed the agent side or the service side into other hosts and I’ll show that in a separate post.

Service

The service’s exe file is “PortBridge.exe” and is both a console app and a Windows Service. If the Windows Service isn’t registered, the app will always start as a console app. If the Windows Service is registered (with the installer or with installutil.exe) you can force console-mode with the –c command line option.

The app.config file on the Service Side (PortBridge/app.config, PortBridge.exe.config in the binaries folder) specifies what ports or named pipes you want to project into Service Bus:


           <
          portBridge 
          serviceBusNamespace
          ="mynamespace" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx" localHostName="mybox">
<hostMappings>
<add targetHost="localhost" allowedPorts="3389" />
</hostMappings>
</portBridge>

The serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret. The serviceBusIssuerName should remain “owner” unless you know why you want to change it. If you don’t have an AppFabric account you might not understand what I’m writing about: Go make one.

The localHostName attribute is optional and when set, it’s the name that’s being used to map “localhost” into your Service Bus namespace. By default the name that’s being used is the good old Windows computer-name.

The hostMappings section contains a list of hosts and rules for what you want to project out to Service Bus. Mind that all inbound connections to the endpoints generated from the host mappings section are protected by the Access Control service and require a token that grants access to your namespace – which is already very different from opening up a port in your firewall. If you open up port 3389 (Remote Desktop) through your firewall and NAT, everyone can walk up to that port and try their password-guessing skills. If you open up port 3389 via Port Bridge, you first need to get through the Access Control gate before you can even get at the remote port.

New host mappings are added with the add element. You can add any host that the machine running the Port Bridge service can “see” via the network. The allowedPorts and allowedPipes attributes define with TCP ports and/or which local named pipes are accessible. Examples:

< add targetHost ="localhost" allowedPorts="3389" /> project the local machine into Service Bus and only allow Remote Desktop (3389)
< add targetHost ="localhost" allowedPorts="3389,1433" /> project the local machine into Service Bus and allow Remote Desktop (3389) and SQL Server TDS (1433)
< add targetHost ="localhost" allowedPorts="*" /> project the local machine into Service Bus and only allow any TCP port connection
< add targetHost ="localhost" allowedPipes="sql/query" /> project the local machine into Service Bus and allow no TCP connections but all named pipe connections to \.\pipes\sql\query
< add targetHost ="otherbox" allowedPorts="1433" /> project the machine “otherbox” into Service Bus and allow SQL Server TDS connections via TCP

Agent

The agent’s exe file is “PortBridgeAgent.exe” and is also both a console app and a Windows Service.

The app.config file on the Agent side (PortBridgeAgent/app.config, PortBridgeAgent.exe.config in the binaries folder) specifies which ports or pipes you want to project into the Agent machine and whether and how you want to firewall these ports. The firewall rules here are not interacting with your local firewall. This is an additional layer of protection.


           <
          portBridgeAgent 
          serviceBusNamespace
          ="mysolution" serviceBusIssuerName="owner" serviceBusIssuerSecret="xxxxxxxx">
<portMappings>
<port localTcpPort="13389" targetHost="mymachine" remoteTcpPort="3389">
<firewallRules>
<rule source="127.0.0.1" />
<rule sourceRangeBegin="10.0.0.0" sourceRangeEnd="10.255.255.255" />
</firewallRules>
</port>
</portMappings>
</portBridgeAgent>

Again, the serviceBusNamespace attribute takes your Service Bus namespace name, and the serviceBusIssuerSecret the respective secret.

The portMappings collection holds the individual ports or pipes you want to bring onto the local machine. Shown above is a mapping of Remote Desktop (port 3389 on the machine with the computer name or localHostName ‘mymachine’) to the local port 13389. Once Service and Agent are running, you can connect to the agent machine on port 13389 using the Remote Desktop client – with PortBridge mapping that to port 3389 on the remote box.

The firewallRules collection allows (un-)constraining the TCP clients that may connect to the projected port. By default, only connections from the same machine are permitted.

For named pipes, the configuration is similar, even though there are no firewall rules and named pipes are always constrained to local connectivity by a set of ACLs that are applied to the pipe. Pipe names must be relative. Here’s how a named pipe projection of a default SQL Server instance could look like:

< port localPipe ="sql/remote" targetHost="mymachine" remotePipe="sql/query"/>

There’s more to write about this, but how about I let you take a look at the code first. I’ve also included two setup projects that can easily install Agent and Service as Windows Services. You obviously don’t have to use those.

[Updated archive (2010-06-10) fixing config issue:]

PortBridge20100610.zip (90.99 KB)

SocketShifter - Network virtualization over the .NET Service Bus

Mon, 06 Jul 2009 16:06:48 GMT

Anyone using the .NET Service Bus should take a good look at the SocketShifter project started by Rob Blackwell and Richard Prodger from AWS in the UK. AWS stands for Active Web Solutions, not for the "other" AWS. The full project is up on Codeplex.

What makes SocketShifter significant is that it takes the network abstraction of SOAP, WS-Addressing, and the Service Bus full circle and layers the very bottom of that stack - plain TCP connections - as a virtualization on top of the the stack. In other words: SocketShifter allows you to create full-fidelity, bi-directional socket connections through the .NET Service Bus.

We've created something very similar to SocketShifter last year (we're using it for a few internal purposes), but haven't made it public so far. I'm glad that the AWS folks built this, so that you get to play with it.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 5: The Queue API for the rest of us

Tue, 07 Apr 2009 06:38:26 GMT

In Parts 3 and 4 of this series I’ve explained the REST protocol for the .NET Service Bus Queue capability. If that looked a little too complicated for your taste and you’d rather clean with a simple API surface, here’s the API that you’ll prefer.

We’ve got two simple classes in the SDK (in the Microsoft.ServiceBus assembly/namespace) that do the job. The QueueManagementClient allows you to create, renew and delete queues in/from the .NET Service Bus namespace:


            1: publicstaticclass QueueManagementClient 
            2: {

            3:    publicstatic QueueClient
CreateQueue(TransportClientEndpointBehavior credential, 
            4:                                          Uri queueUri, QueuePolicy queuePolicy);

            5:    publicstaticvoid DeleteQueue(TransportClientEndpointBehavior
credential, 
            6:                                   Uri queueUri);

            7:    publicstatic QueueClient
GetQueue(TransportClientEndpointBehavior credential, 
            8:                                       Uri queueUri);

            9:    publicstatic QueuePolicy
GetQueuePolicy(TransportClientEndpointBehavior credential, 
           10:                                             Uri queueUri);

           11:    publicstatic DateTime
RenewQueue(TransportClientEndpointBehavior credential, 
           12:                                      Uri queueUri, TimeSpan requestedExpiration);

           13: }

The most striking difference between this API (and the underlying SOAP protocol for which we still owe you some docs) and the REST API/Protocol is that there’s only a single URI to deal with. The WS-Transfer aligned protocol to manage the policy, the “Queue Transfer” protocol used to dequeue messages, and the “Enqueue” protocol to add messages to the queue are all overlaid over the exact same URI.

The only somewhat ugly element here is the TransportClientEndpointBehavior that’s our awkwardly named credentials container. That class was meant to evolve into something else and then we changed our mind at some point, which is how the baby got stuck with that name. We’ll give that a prettier moniker in one of the next CTPs and as part of an overhaul of the .NET Access Control integration into Service Bus.

The function of the methods themselves should be quite obvious. You can create a queue by applying a policy, you can delete a queue, attach to an existing queue, get the latest queue policy, and renew (extend the lifetime) of a queue. For completeness, here’s CreateQueue in context:


            1: Uri queueUri = ServiceBusEnvironment.CreateServiceUri("sb",
solutionName, "/MyQueue/"); 
            2: 

            3: QueuePolicy queuePolicy = new QueuePolicy(); 
            4: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

            5: 

            6: QueueClient client = QueueManagementClient.CreateQueue(credential, queueUri, queuePolicy);

            7: queuePolicy = client.GetPolicy(); //
get effective policy

The QueueClient class allows interaction with a Queue. QueueClient instances cannot be created directly, but must be created via the QueueManagementClient factory.


            1: publicsealedclass QueueClient 
            2: {

            3:     publicvoid DeleteLockedMessage(Message
message); 
            4:     publicvoid DeleteQueue(); 
            5:     public DateTime
GetExpiration(); 
            6:     public QueuePolicy
GetPolicy(); 
            7:     public Message
PeekLock(); 
            8:     public Message
PeekLock(TimeSpan timeout); 
            9:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages); 
           10:     public IEnumerable<Message>
PeekLockMultiple(int maxMessages, TimeSpan timeout); 
           11:     publicvoid Purge(); 
           12:     publicvoid ReleaseLock(Message
message); 
           13:     public DateTime
Renew(TimeSpan requestedExpiration); 
           14:     public Message
Retrieve(); 
           15:     public Message
Retrieve(TimeSpan timeout); 
           16:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages); 
           17:     public IEnumerable<Message>
RetrieveMultiple(int maxMessages, TimeSpan timeout); 
           18:     publicvoid Send(Message
message, TimeSpan timeout); 
           19:     public RouterSubscriptionClient
SubscribeToRouter(RouterClient routerClient, TimeSpan requestedTimeout); 
           20: }

As you can tell, the class is – well – a queue client. It’s using WCF’s Message class as its message abstraction and supports sending messages into the queue, reading messages off the queue in a destructive fashion (Retrieve), and reading messages off the queue using the Peek/Lock pattern which provides resilience against message loss if the message were to be lost in transfer or the receiver fumbled the message. What you’ll also notice is that there are RetrieveMultiple and PeekLockMultiple variants of the retrieval functions which allow for getting more data with fewer network roundtrips.

Again, there should be no surprises using the API. Here’s how you send:


            1: queueClient.Send(Message.CreateMessage(MessageVersion.Default, "Hello",
input), TimeSpan.MaxValue);

and here’s how you do a destructive read:


            1: Message message = queueClient.Retrieve();

and here’s how you use the Peek/Lock pattern:


            1: Message message = queueClient.PeekLock();

            2: string content
= message.GetBody<string>(); 
            3: queueClient.DeleteLockedMessage(message);

There are several examples in the SDK showing how to use the API for Queues. The OnewayQueueSender and SoapHttpQueueSender samples are particularly interesting since they just use WCF channels to enqueue, including all the bells and whistles you get from using a WCF channel. Here’s how the OnewayQueueSender does it:


            1: ChannelFactory<IOnewayChannel> channelFactory = 

            2:      new ChannelFactory<IOnewayChannel>(new NetOnewayRelayBinding(), 
            3:                                         new EndpointAddress(queueUri)); 
            4: channelFactory.Endpoint.Behaviors.Add(userNamePasswordServiceBusCredential);

            5: IOnewayChannel onewayChannel = channelFactory.CreateChannel();

            6: 

            7: string input
= Console.ReadLine(); 
            8: onewayChannel.Hello(new HelloMessage(input));

The logical next question is: So why is the Queue not hooked up to a WCF listener? Answer: That’s what we’ve got “buffered Routers” for. The queue very explicitly provides a ‘pull’ model and the WCF listener would abstract that away and turn it into ‘push’. Routers provide ‘push’ natively and they can embed a Queue. More on Routers in the next set of posts in this series.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 4: The REST Queue Protocol in Code Snippets

Tue, 07 Apr 2009 05:40:58 GMT

After I’ve gone through the dry facts of the REST Queue Protocol in Part 3 of this series, here’s some code to look at. The code snippets are ripped from the HTTP Queue sample we’ve got in the .NET Services SDK and you can get the copy/paste ready code from there.

The most straightforward way to acquire the required hashed security token (see my comments to Stefan Tilkov) using HttpWebRequest looks like this:


            1: staticstring HttpGetAuthenticationToken(string username, string password) 
            2: {

            3:     string tokenUri
= string.Format("https://{0}/issuetoken.aspx?u={1}&p={2}", 
            4:                                     ServiceBusEnvironment.DefaultIdentityHostName, 

            5:                                     username, Uri.EscapeDataString(password));

            6: 

            7:     HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUri);

            8:     using (var
tokenResponse = tokenRequest.GetResponse()) 
            9:     {

           10:         using (var
tokenResponseStream = tokenResponse.GetResponseStream()) 
           11:         {

           12:             byte[]
tokenBody = newbyte[500]; 
           13:             int tokenBodyLength
= tokenResponseStream.Read(tokenBody, 0, 500); 
           14:             return Encoding.UTF8.GetString(tokenBody,
0, tokenBodyLength); 
           15:         }

           16:     }

           17: }

           18:

Once you’ve got the token, you need to pick a place in the Service Bus namespace. We’ve got a built-in helper function in the SDK that sits on ServiceBusEnvironment and which knows all the right incantations:


            1:  Uri queueUri = ServiceBusEnvironment.CreateServiceUri(

            2:                    Uri.UriSchemeHttps, 

            3:                    solutionName, 

            4:                    "/MyHttpQueue/");

Invoked in this way, the method yields the URI https://yourproject.servicebus.windows.net/MyHttpQueue/ so there’s spectacularly little magic to it. The reason why we recommend that you use the method in .NET applications is that we’ve broken everyone’s apps going from the previous CTP to the current CTP due to the namespace restructuring and we’d like to avoid doing that again as we improve the namespace story. I don’t foresee any further change of the magnitude we had this time, however.

To create a new Queue, you first create a Queue policy and then apply it to the Service Bus namespace name. We’ve got a Queue Policy object in the SDK that I’ll use here to keep things short. I’m accepting all the defaults that are spelled out in Part 2 of the series, but explicitly I’m setting the expiration to 1 hour. The Queue Policy XSD schema in the the HTTP Queue sample’s readme document, by the way.


            1: QueuePolicy queuePolicy = new QueuePolicy(); 
            2: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);

            3: queueManageUri = HttpCreateQueue(token, queueUri, queuePolicy);

The code for creating the Queue shouldn’t be very surprising. It’s a POST of an Atom 1.0 entry for which I’m using HttpWebRequest and the WCF 3.5 Syndication API. The SDK sample is using a rather terse version of the ‘correct’ way of doing things. I’m quoting the expanded variant from the Text140 sample that I showed at MIX’09:


            1: publicstatic QueuePolicy
HttpCreateQueue(string token, Uri queueUri, QueuePolicy
policy, 
            2:                                           out Uri
queueSelfUri, out Uri queueHeadUri) 
            3: {

            4:     HttpWebRequest webRequest;

            5:     HttpWebResponse webResponse;

            6: 

            7:     queueSelfUri = null; 
            8:     queueHeadUri = null; 
            9: 

           10:     //
Create a new syndication item and add the queue policy as an extension
           11:     SyndicationItem syndicationItem = new SyndicationItem(); 
           12:     syndicationItem.ElementExtensions.Add(

           13:         new SyndicationElementExtension("QueuePolicy", 
           14:                                         "http://schemas.microsoft.com/ws/2007/08/connect", 
           15:                                         policy));

           16: 

           17:     //
create a new POST request and set the Content-Type to represent an Atom 1.0 entry
           18:     webRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
           19:     webRequest.ConnectionGroupName = "queueclient"; 
           20:     webRequest.KeepAlive = false; 
           21:     webRequest.Method = "POST"; 
           22:     webRequest.Headers.Add("X-MS-Identity-Token",
token); 
           23:     webRequest.ContentType = "application/atom+xml;type=entry;charset=utf-8"; 
           24: 

           25:     //
write the item to the request stream
           26:     using (var
requestStream = webRequest.GetRequestStream()) 
           27:     {

           28:         using (var
textWriter = new XmlTextWriter(requestStream,
Encoding.UTF8)) 
           29:         {

           30:             syndicationItem.GetAtom10Formatter().WriteTo(textWriter);

           31:             textWriter.Flush();

           32:         }

           33:     }

           34: 

           35:     //
get the response
           36:     using (webResponse
= webRequest.GetResponse() as HttpWebResponse) 
           37:     {

           38:         return GetQueueInfoFromResponse(webResponse,
queueUri, out queueSelfUri, out queueHeadUri); 
           39:     }

           40: }

           41: 

           42: static QueuePolicy
GetQueueInfoFromResponse(HttpWebResponse webResponse, Uri queueUri, 
           43:                                                     out Uri
queueSelfUri, out Uri queueHeadUri) 
           44: {

           45:     queueHeadUri = null; 
           46:     queueSelfUri = null; 
           47: 

           48:     if (webResponse.ContentType.StartsWith("application/atom+xml;type=entry",
StringComparison.OrdinalIgnoreCase)) 
           49:     {

           50:         Atom10ItemFormatter atomItemFormatter = new Atom10ItemFormatter(); 
           51:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
           52:         {

           53:             atomItemFormatter.ReadFrom(responseReader);

           54:             //
we found it. let's get the policy and the queue head URI
           55:             foreach (SyndicationLink
queueSelfLink in
           56:                            from link in atomItemFormatter.Item.Links 
           57:                            where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
           58:                            select link)

           59:             {

           60:                 queueSelfUri = queueSelfLink.Uri;

           61:                 break; 
           62:             }

           63: 

           64:             foreach (SyndicationLink
queueHeadLink in
           65:                            from link in atomItemFormatter.Item.Links 
           66:                            where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
           67:                            select link)

           68:             {

           69:                 queueHeadUri = queueHeadLink.Uri;

           70:                 break; 
           71:             }

           72: 

           73:             return atomItemFormatter.Item.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
           74:                  "QueuePolicy", 
           75:                  "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
           76:         }

           77:     }

           78:     else
           79:     {

           80:         Atom10FeedFormatter atomFeedFormatter = new Atom10FeedFormatter(); 
           81:         using (var
responseReader = new XmlTextReader(webResponse.GetResponseStream())) 
           82:         {

           83:             atomFeedFormatter.ReadFrom(responseReader);

           84: 

           85:             //
look for an item whose alternate-link equals the desired queue URI. The alternate 
           86:             //
URI is the 'tail' endpoint by which the queue is mapped into the namespace. We are 
           87:             //
comparing Authority and Path since the scheme might be http or https and we want to
match
           88:             //
either.
           89:             foreach (SyndicationItem
requestedItem in
           90:                          from item in atomFeedFormatter.Feed.Items 
           91:                          from link in item.Links 
           92:                          where 

           93:                            (link.RelationshipType.Equals("alternate",
StringComparison.OrdinalIgnoreCase) && 
           94:                             link.Uri.Authority.Equals(queueUri.Authority, StringComparison.OrdinalIgnoreCase) &&

           95:                             link.Uri.AbsolutePath.Equals(queueUri.AbsolutePath, StringComparison.OrdinalIgnoreCase)) ||

           96:                            (link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) && 
           97:                             link.Uri.Equals(queueUri))

           98:                          select item)

           99:             {

          100:                 //
we found it. let's get the policy and the queue head URI
          101:                 foreach (SyndicationLink
queueSelfLink in
          102:                                from link in requestedItem.Links 
          103:                                where link.RelationshipType.Equals("self",
StringComparison.OrdinalIgnoreCase) 
          104:                                select link)

          105:                 {

          106:                     queueSelfUri = queueSelfLink.Uri;

          107:                     break; 
          108:                 }

          109: 

          110:                 foreach (SyndicationLink
queueHeadLink in
          111:                                from link in requestedItem.Links 
          112:                                where link.RelationshipType.Equals("queuehead",
StringComparison.OrdinalIgnoreCase) 
          113:                                select link)

          114:                 {

          115:                     queueHeadUri = queueHeadLink.Uri;

          116:                     break; 
          117:                 }

          118: 

          119:                 return requestedItem.ElementExtensions.ReadElementExtensions<QueuePolicy>( 
          120:                      "QueuePolicy", 
          121:                      "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault(); 
          122:             }

          123:         }

          124:     }

          125:     returnnull; 
          126: }

The “magic” that isn’t straight up Atom Pub is in lines 10-15 above. I’m attaching the policy object to the SyndicationItem abstraction that will be written out as an Atom 1.0 entry. [I know that copy/paste is difficult using this format – I need to clean up that expanded utility class and will post it in downloadable form within the next few days]

The method returns the effective policy that the Service Bus is using, and yields the management URI for the queue where the Atom entry resides (self-link) as well as the head of the queue as output arguments. The queueUri to which you applied the entry has morphed into the Queue’s tail when the method returns successfully. Having the Queue by its tail, submitting messages into it is very simple. Just do a plain POST. Here we’re just writing a simple string from the local variable input as the entity body and send it.


            1: //
send
            2: HttpWebRequest sendRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest; 
            3: sendRequest.Method = "POST"; 
            4: sendRequest.Headers.Add("X-MS-Identity-Token",
token); 
            5: sendRequest.ContentType = "text/plain;charset=utf-8"; 
            6: using (var
sendStream = sendRequest.GetRequestStream()) 
            7: {

            8:     using (var
writer = new StreamWriter(sendStream, Encoding.UTF8)) 
            9:     {

           10:         writer.Write(input);

           11:         writer.Flush();

           12:     }

           13: }

           14: sendRequest.GetResponse().Close();

For a simple destructive read, you use the ‘queue head’ URI and do a DELETE:


            1: HttpWebRequest dequeueRequest = 

            2:     HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

            3:                           "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
            4: dequeueRequest.ConnectionGroupName = "dequeue"; 
            5: dequeueRequest.Method = "DELETE"; 
            6: dequeueRequest.ContentLength = 0;

            7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
            8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
            9: {

           10:     if (response.StatusCode
== HttpStatusCode.OK) 
           11:     {

           12:         using (var
responseBody = response.GetResponseStream()) 
           13:         {

           14:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
           15:             {

           16:                 string data
= responseReader.ReadToEnd(); 
           17:                 Console.WriteLine(data);

           18:             }

           19:         }

           20:     }

           21: }

For a peek/lock read where you first lock the message and then delete the lock when you want to keep the message, you do a POST on the head and remember the X-MS-Message-Lock header value:


            1: HttpWebRequest dequeueRequest = 

            2:      HttpWebRequest.Create(queueHeadUri.AbsoluteUri+

            3:                            "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest; 
            4: dequeueRequest.ConnectionGroupName = "dequeue"; 
            5: dequeueRequest.Method = "POST"; 
            6: dequeueRequest.ContentLength = 0;

            7: dequeueRequest.Headers.Add("X-MS-Identity-Token",
token); 
            8: using (HttpWebResponse
response = dequeueRequest.GetResponse() as HttpWebResponse) 
            9: {

           10:     if (response.StatusCode
== HttpStatusCode.OK) 
           11:     {

           12:         lockUri = new Uri(response.Headers["X-MS-Message-Lock"]); 
           13:         using (var
responseBody = response.GetResponseStream()) 
           14:         {

           15:             using (var
responseReader = new StreamReader(responseBody,
Encoding.UTF8)) 
           16:             {

           17:                 string data
= responseReader.ReadToEnd(); 
           18:                 Console.WriteLine(data);

           19:             }

           20:         }

           21:     }

           22: }

If you decide to keep the message (i.e. your app didn’t puke processing it), you DELETE the lock, otherwise you do a PUT. The code is practically the same except for the method so I’ll just quote DELETE:


            1: HttpWebRequest deleteLockedMessageRequest = HttpWebRequest.Create(lockUri) as HttpWebRequest; 
            2: deleteLockedMessageRequest.ConnectionGroupName = "lock"; 
            3: deleteLockedMessageRequest.Method = "DELETE"; 
            4: deleteLockedMessageRequest.KeepAlive=false; 
            5: deleteLockedMessageRequest.ContentLength = 0;

            6: deleteLockedMessageRequest.Headers.Add("X-MS-Identity-Token",
token); 
            7: deleteLockedMessageRequest.ContentLength = 0;

            8: deleteLockedMessageRequest.GetResponse().Close();

That’s it. Create, Enqueue, Destructive Read, Peek/Lock Read, and .. oh .. yes.. forgot Delete the Queue. Take the snippet above and swap out the lockUri for the queueSelfUri. Done.

Ah, yes, and please … don’t blame me for HTTP or HttpWebRequest or Clemens’ demo coding style requiring too many lines of code. I could have applied a bit of code-compression here, but I’m intentionally trying not to abstract away too much of the protocol here. I’m generally with you, though. The guys putting together the WCF REST Starter Kit are working on making code like that shrink.

The .NET Service Bus REST protocol for Queues - Some comments, some answers

Tue, 07 Apr 2009 04:14:16 GMT

Stefan Tilkov has several interesting remarks regarding our .NET Service Bus REST Queue Protocol that are worth addressing.

Putting a password in the URI to get an identity token seems to expose information unnecessarily

That’s an area where we know that we’re going to change the protocol. We’ve already labeled that protocol as temporary in the documentation for the PDC CTP and we didn’t get all the pieces in the .NET Access Control service together, yet. Since it’s a HTTPS call, the data doesn’t get exposed on the wire, though.

Queue creation seems fine, even though I feel a little uneasy about wrapping this in an Atom entry

Using Atom 1.0 and the Atom Publishing Protocol as the framework for managing the namespace is very intentional - for several reasons. First of all, it’s a standardized protocol for managing generic lists and the elements in those lists. With that we have a stable and accepted protocol framework and there’s plenty of tooling and framework support around it. That’s worth something. All we need to do is to add some simple extensions – the policies – on top of that stack. Beats having to define, version, and maintain a whole protocol.

On the other hand, Atom seems reasonable considering you get an Atom feed from the queue’s “parent” resource

That’s what I mean. All sorts of tools know how to navigate and display Atom.

Very nice to see the use of link/rel to get to the detailed Queue URIs; it would be even better if the rel values themselves were URIs in an MS namespace

I don’t see much potential for collision here and I would find it odd to have something as simple “self” and “alternate” and then add some unsightly QName for my rel expressions. Simple is good.

Using “alternate” for the tail seems strange

“self” refers to the location where the Atom <entry> resides. “alternate” is what the entry points to. Since the Queue gets mapped into the namespace by “sticking its tail out”, the choice of the alternate link is the simplest possible mapping I could think of.

“The queue’s tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.” The tail does not accept GET. Why not? Can’t think of a reason. Also, I don’t get why any method is allowed. It seems to me a POST/PUT combination or an idempotent POST variant would be the way to go.

The way to look at this is that the Queue’s tail is acting on behalf of the receiver/resource that’ll eventually pick up and process the messages. POST, PUT, DELETE, and BOOYAH are all operations that cleanly map to processing operations and can often be delivered asynchronously with a 202 receipt reply. GET and HEAD don’t make much sense when executed in an asynchronous fashion without getting a reply that’s backed by a response for the receiver. OPTIONS is simply reserved for future use.

“DELETE performing a destructive read” – huh? What happens if I don’t get back a response? How can I retry (as I can’t tell whether I get the same or a different message)? Maybe it’s intended to be used only if you don’t care about losing messages.

DELETE is indeed the dequeue operation variant that you’d use if you are ok with occasional message loss and want to trade transfer reliability for fewer roundtrips.

The POST lock/delete approach, on the other hand, is very nice. Maybe it should be made idempotent, again e.g. using POE

POST lock/delete is the dequeue variant where you are doing the trade the other way around. More reliability bought with an extra roundtrip. In my view, idempotent access to individual messages isn’t much of a practical priority for a competing consumer queue. You’ll get a message or a set of messages to look at under a lock and if you walk away from the message(s), those messages pop back into the queue for someone else to look at. There are obviously scenarios where you want to look at a message sequence in other ways than an ordered queue where you can only get at messages as they appear on the head of the sequence – direct message access and per-message idempotent access matter in those scenarios and we’re looking to give you a capability of that sort in a different kind of messaging primitive.

“The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0)”; although my immediate reaction was to question whether DELETE ever carries a body, the HTTP spec indeed doesn’t say anything about this

One of the great things about working here is that there are all sorts of interesting people around. I’ve discussed the use of DELETE and whether you can provide an entity body in either direction with Henrik Frystyk Nielsen, who works as an architect on the WCF team and is one of the co-authors of HTTP 1.1. Henrik’s stance is that all operations allow entity-bodies unless it’s explicitly forbidden in the spec. I don’t have a better authority to talk to.

“The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue’s head URI” - the wording is worse than the actual approach.

I’m sorry that my writing is so clumsy ;-)

Hang In There. Not Done With Explaining Routers and Queues, Yet :)

Tue, 07 Apr 2009 03:20:24 GMT

I’m far from being done with my blog series on Queues and Routers. Here’s how far I’ve yet gotten: Part 1 – Fundamentals, Part 2 - Queue Policies, Part 3 - Queue REST Protocol. I still owe you the Queue API, the Router Policies, the Router REST Protocol, and the Router API. It’s simply that the planning work for the next milestone slows me down a bit - sorry. If you also want the smaller bits of info, start following me on Twitter in the meantime.

.NET Services March 2009 CTP: Host a Public Website At The Kitchen Table or from a Coffee Shop! No Kidding.

Sun, 05 Apr 2009 19:41:36 GMT

.NET Service Bus Reverse Web Proxy: Click here to download the source

Using the application/service built from the sample linked at the top of this post you can host a publicly discoverable and accessible website or Web service from your Windows notebook or desktop machine from within most network environments without having to open up a port on the firewall, mapping a port on your NAT, or using some type of dynamic DNS service to make the site discoverable. All those essential connectivity features are provided by the .NET Service Bus and with the help of the included sample code.

I’m intentionally not bundling this up as a conveniently installable binary along with a nice configuration UI – that’s not my role here. If you want to grab the code and make it part of a cool personal media sharing app, provide external access to a departmental enterprise app, put a prototype out there for a client to play with, host a web service you want to show off, or or provide an installable version with a nice configuration UI – go ahead.

The attached sample application/service has two key capabilities that I’ve repeatedly been asked for:

a) It is a reverse web proxy that can run either as a console application or as a Windows (NT-) service. The reverse web proxy can sit in front of any web server and forward requests to it. I’ve tested this only with IIS as the backend, but I don’t see a reason why this shouldn’t work with Apache or the Web Server built into some J2EE application server.

b) It is a scripting policy host that projects the crossdomain.xml and ClientPolicyAccess.xml files required by Adobe Flash and Microsoft Silverlight into the root of a .NET Services namespace, permitting cross-domain script access from Flash and Silverlight for all endpoints hosted within the namespace. You can easily adjust the code in the sample to restrict access to particular resources within the namespace.

The fundamental architecture is illustrated in the picture. The web application that you want to project out to the public internet sits on some web server on your machine. “Your machine” may be a desktop machine at home or at work or a notebook in a hotel lobby or an airport on WiFi. As long as you’ve got line-of-sight to the .NET Service Bus and the TCP ports 828 and 818 are available for outbound traffic, you’re good. The reverse web proxy app will map any local HTTP server to a name in the .NET Service Bus and forward the traffic between the .NET Service Bus and the HTTP server. The client (any web browser, but also any HTTP Web Service client) will talk to the .NET Service Bus at the given name, the traffic flows to the reverse proxy on your machine and from there to the HTTP server.

I’m hosting (for a few days) a sample dasBlog site instance at http://clemensv6.servicebus.windows.net/dasblog/. The hosting machine for that blog is one of my personal machines. It’s got a local network address assigned by DHCP, it’s not listed in any NAT mappings, and it’s local Firewall isn’t even open for inbound HTTP traffic.

How to install, build, and run

As a prerequisite you will need three things:

Visual Studio 2008 SP1 with the .NET Framework 3.5 SP1.
A .NET Services project account. The quickest route is to go to http://portal.ex.azure.microsoft.com and click “Sign up”. The approval/provisioning is pretty much instantaneous (plus 20 seconds for the provisioning to run through) once you provide your Windows Live ID. No more access codes.
The .NET Services SDK for the March 2009 CTP. Click here to get it.

Unpack the files, and open ServiceBusReverseWebProxy.sln with Visual Studio 2008. In the ServiceBusReverseWebProxy project, find the app.config file and open it. Here’s where you need to put your project name and password and where you map your sites:


            1: <?xml
version="1.0" encoding="utf-8" ?>
            2: <configuration>
            3:     <configSections>
            4:         <sectionname="reverseWebProxy"
            5:                  type="Microsoft.Samples.ServiceBusReverseWebProxy.ReverseWebProxySection,
ServiceBusReverseWebProxy"/>
            6:     </configSections>
            7:     <!--
Add your .NET Services project account information here -->
            8:     <!--
Create a project at http://portal.ex.azure.microsoft.com/ -->
            9:     <reverseWebProxynetServicesProjectName="!!myproject!!"netServicesProjectPassword="!!mypassword!!"
           10:                      enableSilverlightPolicy="true">
           11:     <pathMappings>
           12:       <addnamespacePath="mysite"localUri="http://localhost/mysite/"/>
           13:     </pathMappings>
           14:   </reverseWebProxy>
           15: </configuration>

Put your .NET Services project/solution name into the netServicesProjectName and the password into netServicesProjectPassword.

Then pick a local HTTP server or site and give it a name in your .NET Service Bus namespace. That mapping is done in the <pathMappings> section. There are a few things that are important to note here:

If your project name were ‘clemensv6’ and you map some local URI to the namespacePath ‘dasBlog’, the resulting .NET Service Bus URI would be http://clemensv6.servicebus.windows.net/dasblog/.
The web application should only emit relative paths for links or, otherwise, should have a way to specify the external host address for links. That means that the web application needs to be able to deal with the presence of a reverse proxy. There is no content-level URL rewriter in this sample that would make any corrections to HTML or XML that’s handed upstream. DasBlog allows you to specify the blog site address as some external address and therefore satisfies that requirements.
Redirects and any other HTTP responses that emit the HTTP ‘Location’ header or any other HTTP headers where URIs are returned are rewritten to map the internal view to the external view.
If you set enableSilverlightPolicy to true, there will be crossdomain.xml and ClientPolicyAccess.xml endpoints projected into the root of your project’s namespace, ie. http://clemensv6.servicebus.windows.net/crossdomain.xml

Build. Run.

By default, the ServiceBusReverseWebProxy.exe application will simply run as a console application. If you use installutil –i ServiceBusReverseWebProxy.exe the application will be installed as a Windows Service. The default identity that it is installed under is ‘NETWORK SERVICE’. In restricted networks with constrained IPSec policies (such as the Microsoft Corporate Network), you may have to use a user account instead. You may also have to use some special Firewall-gateway software such as the ISA Firewall client to allow for outbound access to ports 828 and 818.

The actual application code isn’t really all that complicated. The ‘beef’ is in ReverseWebProxy.cs. What might be surprising here is that this class doesn’t use the WCF Service Model, but is using naked WCF channels for the upstream traffic to .NET Services and it’s using HttpWebRequest for the downstream traffic to the local Web Server. The reason for using channels is that the app is never doing any processing on the messages, so the channel model is the most straightforward and efficient way. The reason for using HttpWebRequest is that you can’t suppress auto-redirects on a WCF HTTP client. Since the stack needs to be completely transparent to redirects so that it’s the browser client up on top that gets redirected instead of someone on the way, I simply couldn’t use a WCF channel downstream. Seems to be one of these edge cases that the WCF team downstairs didn’t think anyone would ever need.

Let me know whether and how this works for you. Share the code, improve it, re-blog, let me know. @clemensv on Twitter, same name @microsoft.com for email.