Yaron Naveh's Web Services 2.0 Blog

I'm an MVP (again!)

2011-07-01T21:58:00.000+03:00

I guess I can skip the traditional email quote by now. I just got the 2011 MVP award in Connected System Developer. Becoming an MVP last year was the beginning of a very interesting year. I'm super exited to get this award for the second time. I'll do my best to justify this recognition.

Binding Box now supprots the Wcf Interop Bindings

2011-06-18T01:03:00.001+03:00

Earlier this week Microsoft had released the Wcf Interop Bindings and VS extension. You can download and try it from here.

Today I am proud to announce that the Wcf Binding Box supports these interoperability bindings.

What is the Wcf Binding Box?
It is an online bindings converter. You give it a binding configuration (e.g. WSHttpBinding) and it returns an equivalent custom binding.

Full explanation is here.

Why do we need it?
Because it's fun :) And also allows to take a working WSHttpBinding and further customize it with settings which it does not directly expose, for example MaxClockSkew.

How the interop bindings relate to this?
Suppose you use the interop bindings to author a Wcf service which WebLogic consumes. You may want to further configure your Wcf service with settings that the WebLogicBinding does not expose. Since the WebLogicBinding internally inherits from WSHttpBinding this is a similar use case to the original purpose of the binding box.

Example

Put this WebSphere binding as the input in the binding box:

and this is the custom binding output:

Check out the binding box here.

Test drive the shiny new Wcf interop bindings

2011-06-14T23:31:00.000+03:00

wcf.codeplex.com is the place where most of the wcf action happens at these days. If you have been following it recently you have seen a lot of activity around Rest and Http. As of yesterday Soap officially joins the codepex party. Microsoft has just released the WCF Express Interop Bindings - a new Visual Studio extension for Soap web services interoperability. If you use Wcf this matters to you!

What did Microsoft release yesterday?

Web services interoperability is always a pain. When security is involved it is usualy more then a casual 'oouch'. Yes, WsHttpBinding has a specific permutation of settings which can interoperate with Oracle web logic. And I know a lot of people who have tried to find that permutation in a brute force manner. Mostly doing this is a waste of time which we prefer to invest in more productive areas.

So here's the idea behind yesterday's shipping: We now have a new binding, WebLogicBinding, which only allow us to configure settings which are interoperable with web logic. So all settings are interoperable! We also have bindings for web sphere, axis2 (wso2) and metro (wsit / glassfigh / tango).
In addition we got a nice wizard on top of Visual Studio's new project dialog which allows us to easily author interoperable services using these bindings.

But don't we already have WS-Policy for interoperability?

WS-Policy helps clients to generate proxies which complies with the service requirements as expressed by the Wsdl. The express bindings solves a prior problem: How to write from the first place a service which a specific client platform can support? Once we write such a service its Wsdl will contain the WS-Policy pixy dust so that the client can auto-configure itself.

Nice... I'll take two!
You can take four: MetroBinding, WebSphereBinding, WebLogicBinding and Wso2InteropBinding.
Take them from here.

Tutorial - WCF and Metro interop

Let's see why web services interoperability just got a whole lot easier.
We'll create a WCF service with mutual x.509 certificates in the message level and consume it with a Metro client.

1. Prepare the environment
You need VS 2010 and the express bindings. After you extract the bindings zip simply execute bin\Microsoft.ServiceModel.Interop.Extension.vsix which will install it on VS.

2. Create a new service

In VS create a new project. Note how the Wcf node now contains a new project type "Express Interop Wcf Service Application":

Choose that project type.

3. Configure the express binding wizard

A few moments after creating the project you will see the wizard.
First choose the platform our clients will use - Metro, this time.

Now we need to configure our security requirements. Choose "mutual certificate" which means both client and server will present an x.509 certificate in the message level. It also implies encryption and digital signature (in this case). To keep it simple we omit the secure conversation.

Next in the advanced settings use the Basic128 algorithm since it is the one Metro supports by default (for Basic256 a patch needs to be applied).

Finally configure the certificate.

I recommend to use this certificate (password: adminadmin):

Now run the service. This is a web site project so it will open the documentation page with the Wsdl link. Make sure to have the Wsdl url handy since we will use it in a moment.

Now we want to configure Metro.

1. Set up the environment

You should have NetBeans 7 (or higher), though NetBeans 6.7 also worked for me.

2. Create a new project of type Java Application:

Any of the default settings are fine:

3. Right click the package in the project view and add a new "web service client":

Now is a good time to paste that Wsdl url:

4. The service reference is now in the project view so right click it and edit the Web Service Attributes (similar to the Wcf configuration... just very different :):

5. This step is a workaround if your NetBeans version ships with Metro 2.0 (which is the case for NetBeans 7). See below how to know if you need it.

We can see that Metro had automatically identified that client and server certificates are required. This was due to the WS-Policy in the Wsdl.

Before we continue we need to do some trick. NetBeans 7 ships with Metro 2.0 which has a bug with certificates. In favor of those who reach this post via a search engine this is the error message:

java.lang.NullPointerException
...
at com.sun.xml.ws.security.impl.policy.CertificateRetriever.digestBST (CertificateRetriever.java:136)

To solve this you need to download Metro 2.1 (or higher). For now just extract it to some folder.

Now as part of this workaround check the "use development defaults" drop down in the quality attributes dialog you opened in step 4. Also approve any message you are prompt with.

Click Ok to close the dialog.

In the project pane expand the libraries node. It should look like this:

This workaround applies to version 2. If you see another version (even smaller) no need for this. What you need to do is delete all the references to Metro jar files (don't delete the jdk though). Instead of them right click the "libraries" node, choose "add jar/folder..." and choose the jar files in metro\bin folder from the metro 2.1 zip you just extracted. Add all jar files in that folder. The libraries node will now look like this:

6. We are now ready to do the actual configuration. Open again the web service quality attribute form as you did in step 4. Uncheck the "use development defaults" check box. Now configure the keystore and trust store. I recommend to use this java key store file:

Open "keystore..." and "Truststore.." each in its turn and do the below.
Set the path to the .jks file you extracted form the certificates file above, set the password to "adminadmin", and click "load alias". The alias for the key store is xws-security-client and for the trust store is xws-security-server.

7. Now we need to write the client code.

Since most of my readers are .Net developers let's see if we can pull this one out without any Java coding at all.

Drag the GetData node from the project pane to the main() method. It should now look like this (depending on the netbeans version):

public static void main(String[] args) {

try { // Call Web Service Operation
     org.tempuri.Service service = new org.tempuri.Service();
     org.tempuri.IService port = service.getMetroBindingIService();
     // TODO initialize WS operation arguments here
     java.lang.Integer value = Integer.valueOf(0);
     // TODO process result here
     java.lang.String result = port.getData(value);
     System.out.println("Result = "+result);
}
catch (Exception ex) {

}

}

if you use Netbeans 7 it will only generate a method so you would need to add code that calls it:

public static void main(String[] args) {
try
{
     String s = getData(2);
     System.out.println("result is" + s);
} catch (Exception ex) {
     System.out.println(ex.getMessage() + ex.getStackTrace().toString());
}
}

private static String getData(java.lang.Integer value) {
   org.tempuri.Service service = new org.tempuri.Service();
   org.tempuri.IService port = service.getMetroBindingIService();
   return port.getData(value);
}
}

7. Now run the application (F6).

Here is the output:

the result is: 2

Web services interoperability was never easier!

Cannot resolve KeyInfo for unwrapping key

2011-05-08T17:44:00.000+03:00

With web services sometimes your client is able to receive a good response from the server but your client will still throw exception due to some policy violation. With wcf / mutual authentication the following error can appear:

Cannot resolve KeyInfo for unwrapping key: KeyInfo 'SecurityKeyIdentifier
(
IsReadOnly = False,
Count = 1,
Clause[0] = X509IssuerSerialKeyIdentifierClause(Issuer = 'CN=MyCert', Serial = '-903515464456238801534567116928')
)
', available tokens 'SecurityTokenResolver
(
TokenCount = 0,
)

This error usually means that the server had digitally signed its response using an unexpected certificate. The expected certificate is the one which the client has configured as the server certificate and have possibly used to encrypt the message with.

So as with many of the security interoperability problems, you should verify that you use the correct certificate on both sides of the wire.

Anyone else having problems with Google Calendar?

2011-05-08T00:56:00.001+03:00

Update (july 11): Fixed!

I'll start by saying that google calendar is a great product which I was happy to use for quite a while. Not any more, it seams. For some reason I intermittently don't get email notifications about scheduled events. I have first noticed that over a year ago and assumed it is a one off glitch. I have then noticed it in many more occasions and since last week I don't get notifications at all.

I mainly use gcal as a "send myself a future email reminder" application. While it does not contain any important appointments, I do have there information about birthdays, random arrangements, sites without rss which I periodically visit etc. And while managing to live without these unsent reminders makes me wonder if I had ever needed them anyway, it is still annoying.

This issue appears in several gcal forum threads, has a special troubleshooting page, and is actually a known issue. The last link says "We're currently investigating reports from users who indicate that they're not receiving email notifications for their events". How many users? What is the current status?

Just out of curiosity - does anyone else experience missing email notifications from google calendar?

Wcf with WS-Addressing March 2004

2011-05-06T22:09:00.000+03:00

Wcf supports two WS-Addressing versions – the August 2004 draft and the actual standard v 1.0. There is another wsa version which is used by some soap stacks (who said wse 2?) – the march 2004 draft. To turn this fact to a more practical problem, you might need to write a wcf client to a (non Wcf) service which expects a request like this:

<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing">

<Header>

    <wsa:Action>http://myAction</wsa:Action>
    <wsa:MessageID>uuid:5024616c-d0r2-56h3-bjj7-ab10p89eee63</wsa:MessageID>
    <wsa:ReplyTo>
      <wsa:Address>http://schemas.xmlsoap.org/ws/2004/03/addressing/role/anonymous</wsa:Address>
    </wsa:ReplyTo>
    <wsa:To>http://server/Calc</wsa:To>
...

(note the bold wsa version – wcf cannot emit it).

One straight forward way to do this with wcf is to push these headers to the soap using a message inspector. But what if you also need to sign the wsa headers with a digital signature?

In this case you need to write a custom behavior which push the wsa headers to the IncomingSignatureParts property. The whole code looks like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Security;
using System.Text;
using System.Xml;
using System.Xml.Linq;
using MyApplication.ServiceReference1;

namespace MyApplication
{

    public class SignMessageHeaderBehavior : Attribute, IEndpointBehavior
    {

        string action;

        List<XmlQualifiedName> headers;

        public SignMessageHeaderBehavior(List<XmlQualifiedName> headers, string action)
        {

            this.headers = headers;

            this.action = action;

        }

        public void Validate(ServiceEndpoint endpoint)
        {

        }

        public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
        {
            var requirements = bindingParameters.Find<ChannelProtectionRequirements>();

            foreach (var h in headers)
            {
                var part = new MessagePartSpecification(h);
                requirements.IncomingSignatureParts.AddParts(part, action);
            }
        }

        public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
        {

        }

        public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
        {

            clientRuntime.MessageInspectors.Add(new AddHeaderMessageInspector());

        }

    }

    [DataContract(Namespace="http://schemas.xmlsoap.org/ws/2004/03/addressing")]
    public class ReplyToHeader
    {
        [DataMember]
        public string Address { get; set; }
    }

    public class AddHeaderMessageInspector : IClientMessageInspector
    {

        public object BeforeSendRequest(ref Message request, IClientChannel channel)
        {

            request.Headers.Add(MessageHeader.CreateHeader("Action", "http://schemas.xmlsoap.org/ws/2004/03/addressing", "http://myAction"));
            request.Headers.Add(MessageHeader.CreateHeader("MessageID", "http://schemas.xmlsoap.org/ws/2004/03/addressing", "uuid:5024616c-d0r2-56h3-bjj7-ab10p89eee63"));
            request.Headers.Add(MessageHeader.CreateHeader("ReplyTo", "http://schemas.xmlsoap.org/ws/2004/03/addressing", new ReplyToHeader() { Address = "http://schemas.xmlsoap.org/ws/2004/03/addressing/role/anonymous" }));
            request.Headers.Add(MessageHeader.CreateHeader("To", "http://schemas.xmlsoap.org/ws/2004/03/addressing", "http://server/Calc"));


            return request;

        }

        public void AfterReceiveReply(ref Message reply, object correlationState)
        {

        }

    }

    class Program
    {

        static void Main(string[] args)
        {

            var c = new SimpleServiceSoapClient();

            var headers = new List<XmlQualifiedName>();
            headers.Add(new XmlQualifiedName("Action", "http://schemas.xmlsoap.org/ws/2004/03/addressing"));
            headers.Add(new XmlQualifiedName("MessageID", "http://schemas.xmlsoap.org/ws/2004/03/addressing"));
            headers.Add(new XmlQualifiedName("ReplyTo", "http://schemas.xmlsoap.org/ws/2004/03/addressing"));
            headers.Add(new XmlQualifiedName("To", "http://schemas.xmlsoap.org/ws/2004/03/addressing"));

           c.Endpoint.Behaviors.Add(
                new SignMessageHeaderBehavior(headers, "http://tempuri.org/EchoString"));

            c.EchoString("123");

        }
    }
}

svcutil.exe with a wsdl on the local disk

2011-04-23T13:57:00.001+03:00

svcutil.exe is a command line to generate wcf proxies from wsdl files (and more).

for example the following command:

$> svcutil http://localhost/MyServices/Service.svc?WSDL

will generate Service.cs which is the proxy you can add to your project.

Sometimes the wsdl file is not on a url but on the local disk. In theory this should work:

$> svcutil c:\services\calc.wsdl

In practice this would work only if the wsdl does not reference other wsdl or xsd files (even if their relative reference is correct). The same would work with "add service reference" in VS so not sure why it fails here. One solution is to explicitly specify all the referenced schema files in the command:

$> svcutil" c:\services\calc.wsdl" "c:\person.xsd c:\units.xsd"

This usually works but requires manual work and is not always desired when a large number of references is used.

An alternative would be to upload the wsdl and the references to a web server and svcutil from there. If you have iis on your dev machine this is actually pretty simple:

1. copy the wsdl root folder to a subfolder under c:\inetpub\wwwroot
2. run

$> svcutil http://localhost/root/calc.wsdl

That's all, no need to specify all files. svcutil works well from url locations. Do not forget to remove the wsdl folder after so your iis folder would stay clean.

...and, we're back!

2011-04-22T22:32:00.001+03:00

This post violates at least 4 of the blogging cliches. Read at your own risk!

You might have noticed I've disappeared for a while. The reason is that I have been practically leaving on an airplane in the last two months beeing over 80 hours in flights.

In early march I have visited Redmond for the MVP global summit. I've met people which so far I've only seen in the cover of their books. I've met some of my favorite bloggers and readers. And even though I'm a Connected Systems MVP I took a picture with the Biztalk MVP's (thanks to leonid who took the photo):

I have then visited customers in many states in the US. I never regretted for taking that extra coat and clothes because some places were damn cold! Still I had a lot of fun, met some super smart people, and also had time to visit beautiful places:

Shortly after I came back home I got my 5th star in the MSDN forums:

I'm very proud for being able to help so many people. I am going to continue and participate in these forums not only because I love to help but also since I learn a lot on the way.

More lessons learned on Windows Azure pricing

2011-02-25T13:58:00.000+02:00

My Bart Simpson's guide to windows Azure has been highly successful. From Twitter to the post comments to my mail box I got a lot of good feedback.

Here are some insights you might want to know:

Extra small instances might not be included in your free Azure account.
David Pallmann was the first to notice that some of the special Azure offers (like the MSDN one) actually have some small letters:

**Extra small compute instances are available in beta and are billed separately from other compute instance sizes.

This practically means you pay for these instances from day 1! Here is my account:

So you should check your account details and determine if extra small instances work for you.

I also recommend you check out David's Hidden Costs in the Cloud series which has some important insights.

Hot news: Azure trial with extra small instances
A few days ago Microsoft has started to offer an introductory offer which includes 750 free monthly hours of x-small instances (until June). Check it out.

Also thanks to David Makogon I've fixed an error in the config sample to configure extra small instances.

Bart Simpson's guide to Windows Azure

2011-02-19T15:17:00.002+02:00

The original name of this post was "a poor developer's guide to windows azure" but then I found the Bart Simpson's Chalkboard Generator:

My Azure story begins back in PDC08 where Microsoft announced a community preview of Windows Azure which includes a free subscription for a limited period. I took advantage of this to develop the Wcf binding box (which btw got some good reviews). A few months after, the preview has ended and my account became read only. I was not too bothered by it as I had other things in mind. A few weeks ago I had a crazy idea to build a wsdl2-->wsdl1 converter. The most natural way to do it was to create an online service. But my azure trial is already in freeze, and I did not want to pay a hosting service just to host a free contribution I make for the community. What could I do? I then remembered that I (and my contest winners) have a special MSDN premium Azure offer. And this is how wsdl2wsdl came to life. Veni, vidi, vici? Oh my...

A few days after going on air I get this email from Microsoft:

Your Windows Azure Platform Usage Estimate - 75% of Base Units Consumed‏

This e-mail notification comes to you as a courtesy to update you on your Windows Azure platform usage. Our records indicate that your subscription has exceeded 75% of the compute hours amount included with your offer for your current billing period. Any hours in excess of the amount included with your offer will be charged at standard rates.

Total Consumed*: 592.000000 Compute Hours
Amount included with your offer: 750 Compute Hours
Amount over (under) your monthly average: -158.000000 Compute Hours

Let's see... I should get 750 compute hours / month, a month has 31 days (a worse case analyses), 750 / 31 > 24 which means I should have more than 24 complimentary compute hours per day. How could they run out so fast?

Bart Simpson's Azure Rule #1:

As a developer, it made too much sense to develop the binding box and wsdl2wsdl in a separate visual studio solutions. This yields two separate azure hosted services:

And this yields two separate bill items per day:

This means I was not paying for 24 compute hours per day, I was paying 24*2!

Now you may say RTFM. But nobody does it. Not when we download some open source library from the web, and not when we upload something in the other direction.

But why did the bill had 4 itmes and not two?

Bart Simpson's Azure Rule #2:

Not only did I had two hosted services online, but I also had two environments for each - staging and deployment. You pay for what you get:

2 services * 2 deployments = 4 * 24 hours a day = 96 hours a day!

Bart Simpson's Azure Rule #3:

I was on my way to a Chapter 11 when I decided to panically press the stop button on my unintended deployments:

However this does not reduce costs:

Suspending your deployment will still result in charges because the compute instances are allocated to you and cannot be allocated to another customer.

Remember: Always delete deployments you do not want to pay for. Suspending / stopping them still results in charges.

Bart Simpson's Azure Rule #4:

For applications with very small needs an extra small instance should be enough. It can save you costs by up to 60%! Configuring it is very easy:

Conclusion

Homer Simpson once said "Trying is the first step towards failure." In my case it was the first step to this blog post and to my first two weeks Azure bill:

I'll end the month on around 50$ which is not too bad for a commercial site. But for a contribution to the community and my fun? Oh my...

Weather you are an independent developer or a Fortune 5000 company - know the Azure pricing model and how your account fits in.

Wcf: Keyset does not exist

2011-02-13T19:20:00.060+02:00

When using X.509 certificates with Wcf the below error may appear:

System.Security.Cryptography.CryptographicException: Keyset does not exist
ArgumentException: The certificate 'CN=MyCert' must have a private key that is capable of key exchange. The process must have access rights for the private key.

95% of the time this means that the certificate which the server/client use either does not have a private key or the Wcf host process does not have permissions to the key.

No private key
This case applies when the certificate is expected to have a private key, e.g. the server private certificate when defined on the server side, and not the server public when defined on the client. To check if the certificate has a private key follow these steps:

1. start-->run-->"mmc"

2. file-->add remove snap in...

3. double click "certificates" in the list

4. Choose "My user account" if the certificate is located in the current user store, or "Computer account" if located on the local machine store. If you are unsure you can repeat the process twice each time with a different choice.

5. click Finish + Ok

6. now expand the tree to the correct store and when you see the certificate double click it. Then check if it has the little key icon on it. If it does not then you did not import it with its private key (or got the wrong cert).

No permissions
Even if the certificate has a private key, it still does not mean all users on the machine have access to it. One common gotcha is to give access to the admin (or logged in user) but forget that IIS usually runs under another user account. This may cause a code to work correctly under an interactive user but to fail under IIS or any windows service. One way to check if this is the case is אם give the user full permissions to the key (temporarily!).

How to give permissions to a key?
the hard way is using WinHttpCertCfg.exe (details ,download):

winhttpcertcfg -g -c LOCAL_MACHINE\My -s CN=WSE2QuickStartServer -a SomeUser

Another way is using some gui utility and WseCertificate2.exe is a good one:

1. install the Wse2 sdk

2. run C:\Program Files\Microsoft WSE\v2.0\Tools\Certificates\WseCertificate2.exe

3. choose the certificate using the location / store drop down lists and the "open certificate" button.

4. click the "view private key file properties..." button on the bottom.

5. depending on your OS version, grant permissions for the user you want.

If all this did not help ten make sure that when have you installed the certificate you checked the "mark this key as exportable" checkbox:

Sometimes these permissions are cached so you can also restart IIS (and maybe even the PC). And as always with certificate, when you're already pulling out your hair it's time to uninstall all certificates and start all over again.

Wcf support for Wsdl2

2011-02-12T12:28:00.003+02:00

Last time I introduced wsdl2wsdl, an online wsdl2-->wsdl1 converter.
Today I am proud to announce svcutil2 - a Wcf proxy generator for Wsdl2. svcutil2, like the original svcutil, generates Wcf proxies from Wsdl2 documents. This is a huge accelerator for web services interoperability. See related discussion here.

svcutil2 is fully open sourced in CodePlex.

How to generate Wcf clients from Wsdl2 documents?
1. Download the latest version of svcutil2.exe from CodePlex

2. Open the VS command console or otherwise make sure the original svcutil.exe is in the current path (usually located in C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin)

3. Use svcutil2 exactly like you use svcutil:

$> svcutil2.exe http://webservices20.cloudapp.net/wsdl2wsdl/wsdl/simple2.wsdl

you can also work with Wsdl's from file system or use any of the svcutil available flags.

Downgrade! Convert Wsdl 2.0 to Wsdl 1.1 (Wsdl2Wsdl)

2011-02-05T23:53:00.003+02:00

Wsdl2Wsdl is Here!

Wsdl 2.0 never fulfilled its promises: It did not replace Wsdl 1.1 and soap stacks rarely support it. It did not became the Metadata standard for Rest based web services. And in an era where a service metadata can be summed up with a resource uri, Wsdl 2.0 is barely looked at as a simplification. All this is hardly Wsdl 2.0 fault. It's the timing. The backlash against Soap/Wsdl based services was far behind its point of no return in the period where Wsdl 2.0 was expected to get high adoption. It is not too risky to bet that Wsdl 2.0 will never be implemented by existing soap stacks, and Wcf is no exception.

And the sky is blue. So?
While service authors may elegantly ignore Wsdl 2, if you're on the consumer side you might need to consume a service which metadata is a Wsdl 2 document. Most chances are that your client stack does not support code generation from this version of Wsdl. This is an annoying interoperability problem.

Wsdl2Wsdl
I have written an online Wsdl 2 --> Wsdl 1 converter. While there are a number of Wsdl1 --> Wsdl 2 converters available, I have yet to find one in the opposite direction.
Wsdl2Wsdl provides you a url which dynamically converts the Wsdl per demand. This means you always get the live version of the wsdl.

Can I run Wsdl2Wsdl on my premise?
The current version of Wsdl2Wsdl is web based so if your Wsdl has some secrets / IP you should take this into consideration. I plan to open source it and ship an on premise version. You are welcome to contact me if you need this urgently.

Where is Wsdl2Wsdl?
It's hosted on my Azure account:

http://webservices20.cloudapp.net/wsdl2wsdl.html

Drop me a mail with any issue or feedback.

Utility to export X.509 certificates

2011-02-02T22:47:00.000+02:00

The number of X.509 formats \ stores can be overwhelming: pfx, p12, pem, jks, windows store and more. When working on multi platform projects exporting the certificate from one format \ store to the other is essential. Traditionally the way to do it was with openssl. While there is nothing wrong with that, you usually forget how to use it a minute after you do, and need to learn it again the next time.

Over a year ago travis had published a super relevant utility to export certificates. It is really easy to use and prevents that recurring learning curve with openssl.

Xml Digital Signature: Signing the KeyInfo

2011-01-30T23:36:00.000+02:00

Frederic Vidal had shown me recently a nice trick with Xml digital signatures. Suppose you want to add a signature which looks like this:

<ds:Signature xmlns:ds='http://www.w3.org/2000/09/xmldsig#' Id='Signature001'>
   <ds:SignedInfo>
     <ds:CanonicalizationMethod Algorithm='http://www.w3.org/TR/2001/REC-xml-c14n-20010315' />
     <ds:SignatureMethod Algorithm='http://www.w3.org/2000/09/xmldsig#rsa-sha1' />
     <ds:Reference URI=''>
       <ds:Transforms>
         <ds:Transform Algorithm='http://www.w3.org/2000/09/xmldsig#enveloped-signature' />
       </ds:Transforms>
       <ds:DigestMethod Algorithm='http://www.w3.org/2000/09/xmldsig#sha1' />
       <ds:DigestValue>sXe2PnaG...</ds:DigestValue>
     </ds:Reference>
     <ds:Reference URI='#KeyInfo001'>
       <ds:DigestMethod Algorithm='http://www.w3.org/2000/09/xmldsig#sha1' />
       <ds:DigestValue>ZOS23PQ9TcDu+G...</ds:DigestValue>
     </ds:Reference>
   </ds:SignedInfo>
   <ds:SignatureValue>jTLX0/8XkY2aCte7...</ds:SignatureValue>
     <ds:KeyInfo Id='KeyInfo001'>
       <ds:X509Data>
         <ds:X509Certificate>E3wdSY4n7MgUmJzMIGfMA0...</ds:X509Certificate>
     </ds:X509Data>
   </ds:KeyInfo>
</ds:Signature>

The interesting part is the second reference (in bold) – the signature signs the KeyInfo (#KeyInfo001), which is part of the signature element itself. The regular api to add reference to a signature is this:

var reference = new Reference();
reference.Uri = "#KeyInfo001";

However it will not work here since it will look for an element with this ID in the signed document (e.g. soap envelope). However the ID is inside the signature element itself, which is still not a part of the document because it was not create yet.
Frederic had found a nice trick: Inherit from SignedXml and override the default logic to find the references such that it will search in the signature itself (which is the base class).

public class CustomIdSignedXml : SignedXml
{
   public CustomIdSignedXml(XmlDocument doc) : base(doc)
   {
     return;
   }

   public override XmlElement GetIdElement(XmlDocument doc, string id)
   {
     if (String.Compare(id, this.KeyInfo.Id, StringComparison.OrdinalIgnoreCase) == 0)
       return this.KeyInfo.GetXml();
     else
       return base.GetIdElement(doc, id);
   }
}

This is applicable to web services, since many soap stacks will not allow to customize them to sign the KeyInfo, and if this is required you would need to sign the message like this yourself.

gSoap and Wcf Routing Services are not friends

2011-01-29T23:43:00.002+02:00

One common patterns with web services is the router service. It can hide routing logic from the client or help with load balancing. Wcf 4 ships with libraries and samples to help build such a router very quickly.

Stephen Liedig had notified me recently on a problem which happens when a gSoap client calls a Wcf router. gSoap is a very popular CPP web services stack.

This was the deployment:

The gSoap client was sending this message to the router:

<soap:Envelope xmlns="http://tempuri.org/" xmlns:p="http://myNs/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soap:Body>
    <AddUser>
      <user xsi:type="p:User">
        <name>yaron</name>
      </user>
    </AddUser>
</soap:Body>
</soap:Envelope>

If the router would call the service using the http endpoint, everything would work. If it used the tcp endpoint, it could send the message, but the service implementation would never be called and the service would report this exception:

The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://tempuri.org/:user. The InnerException message was 'Element 'http://tempuri.org/:user' contains data of the ':User' data contract. The deserializer has no knowledge of any type that maps to this contract. Add the type corresponding to 'User' to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding it to the list of known types passed to DataContractSerializer.'. Please see InnerException for more details.

Analysis

We need to ask two questions: Why this error happens? Why not with Http?

First we must take a look in the message that the router sent to the service. Here is how it appears in the Wcf log:

<soap:Envelope xmlns="http://tempuri.org/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soap:Body>
    <AddUser>
      <user xsi:type="p:User">
        <name>yaron</name>
      </user>
    </AddUser>
</soap:Body>
</soap:Envelope>

Something is missing – the “p” prefix declaration (xmlns:p=http://myNs/) which was present in the message form the client to the router does not appear. This makes this message invalid as it references the undeclared prefix "p" in the "p:User" derived type attribute. But why did the router sent an invalid message when the client sent it a good one? And why not with Http?

Let’s take a look at how Wcf routes messages. We can use the reflector to inspect System.ServiceModel.Routing.SoapProcessingBehavior+SoapProcessingInspector.MarshalMessage():

internal Message MarshalMessage(Message source, Uri to, MessageVersion targetVersion)
{
    Message message;
    MessageVersion version = source.Version;
    if ((version == targetVersion) && !RoutingUtilities.IsMessageUsingWSSecurity(understoodHeaders))
    {
        ...
        return source;
    }
    ...
    else
    {
        XmlDictionaryReader readerAtBodyContents = source.GetReaderAtBodyContents();
        message = Message.CreateMessage(targetVersion, headers.Action, readerAtBodyContents);
    }

    this.CloneHeaders(message.Headers, headers, to, understoodHeadersSet);
    return message;
}

Wcf uses this logic:

If the input message (from client) and output message (to server) have the same version, and no security is involved, then take the input as is and send it. This explains why the Http case works.
Otherwise take the input message body and copy it to a new message. Then copy custom user headers.

Since the “p” prefix is not defined under the body but under the root “envelope” element, this prefix is not sent which makes the message invalid. This explains the netTcp bug.

Why Wcf behaves in this way?

Dismissing this behavior as a bug will miss an important discussion. Consider the naïve fix: Parse each attribute under the body and search for the something:something pattern. Since the router has no information about the message semantics it has no way to know if the pattern relates to a prefix or is just a string which looks like this. Moreover doing this would be a huge performance hit, especially with big messages.

What should be the correct behavior?

In order to take both the functional and performance requirements into account, I would recommend the following approach:

Copy all namespace declarations from the root Envelope and Body elements of the original message into the corresponding elements of the new one (including the default namespace).
Make sure not to overload the previous prefixes with new definitions in both elements.

Yes, I can think of a few edge cases where this scheme fails. But a more comprehensive solution would cost in performance.

Conclusion

As it stands now, the default gSoap client fails to call the default Wcf router when protocol bridging is used, which is a real interoperability problem.

And Just to clarify, this web service does not define any derived (=inherited = known) types. For some reason the gSoap default message generation logic uses the xsi:type even on base types. This is not forbidden since it uses the correct type name, although there is no real reason to do it. Anyway this makes this case quite common and not limited to services with derived types.

And, yes, this is one of the reasons not everyone likes Xml.

Security interop gotcha: Empty signature transformation

2011-01-14T22:14:00.001+02:00

Markó had challenged me recently with a security interoperability case between Wcf and WSIT (metro). A mutual X.509 protection was involved, and the Java server was returning the following soap fault:

com.sun.xml.wss.XWSSecurityException: Encryption Policy verification error: Looking for an Encryption Element in Security header, but found com.sun.xml.wss.impl.policy.mls.SignaturePolicy@3657517

I have not seen this error before, but I have followed the book and asked for a sample working soap message:

<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsse11="http://docs.oasis-open.org/wss/oasis-wss-wssecurity-secext-1.1.xsd" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:xenc="http://www.w3.org/2001/04/xmlenc#" xmlns:exc14n="http://www.w3.org/2001/10/xml-exc-c14n#">

<S:Header>

    <To xmlns="http://www.w3.org/2005/08/addressing" wsu:Id="_5006">http://server.com</To>

    <Action xmlns="http://www.w3.org/2005/08/addressing" wsu:Id="_5005">someAction</Action>

    <ReplyTo xmlns="http://www.w3.org/2005/08/addressing" wsu:Id="_5004">

      <Address>http://www.w3.org/2005/08/addressing/anonymous</Address>

    </ReplyTo>

    <MessageID xmlns="http://www.w3.org/2005/08/addressing" wsu:Id="_5003">uuid:203aa9ea-eea6-4f96-b0b2-16575411e9a2</MessageID>

    <wsse:Security S:mustUnderstand="1">

      <wsu:Timestamp xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope" wsu:Id="_3">

        <wsu:Created>2010-12-22T14:25:10Z</wsu:Created>

        <wsu:Expires>2010-12-22T14:30:10Z</wsu:Expires>

      </wsu:Timestamp>

      <wsse:BinarySecurityToken xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3" EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary" wsu:Id="uuid_87b326f6-54fe-42d5-a3a5-aea7cb12f66b">PO7sWakqK...</wsse:BinarySecurityToken>

      <xenc:EncryptedKey xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope" Id="_5002">

        <xenc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p" />

        <ds:KeyInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="KeyInfoType">

          <wsse:SecurityTokenReference>

            <wsse:KeyIdentifier ValueType="http://docs.oasis-open.org/wss/oasis-wss-soap-message-security-1.1#ThumbprintSHA1" EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">TzX5OGaS9Ftsw1t+eGyfBmJblWc=</wsse:KeyIdentifier>

          </wsse:SecurityTokenReference>

        </ds:KeyInfo>

        <xenc:CipherData>

          <xenc:CipherValue>WJU7hIO...</xenc:CipherValue>

        </xenc:CipherData>

      </xenc:EncryptedKey>

      <xenc:ReferenceList xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope">

        <xenc:DataReference URI="#_5008" />

      </xenc:ReferenceList>

      <ds:Signature xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope" Id="_1">

        <ds:SignedInfo>

          <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

            <exc14n:InclusiveNamespaces PrefixList="wsse S" />

          </ds:CanonicalizationMethod>

          <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#hmac-sha1" />

          <ds:Reference URI="#_5003">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>q9TWoUKb25xlel3AIA8bFUkl0hw=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#_5004">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>5Ab1ebo4/FraGgck/A8iDx1J9+I=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#_5005">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>ipPKrD6igfYF3tC10tsurnoHSks=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#_5006">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>cMG2PvlZKNLoAWehtNhxruuRl9Y=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#_5007">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>rRhCJKKcNv2gfh+Hpi62wDEirbE=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#_3">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="wsu wsse S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>OUm9PNmaQgH1CQ4JitIqVX5eY+g=</ds:DigestValue>

          </ds:Reference>

          <ds:Reference URI="#uuid_87b326f6-54fe-42d5-a3a5-aea7cb12f66b">

            <ds:Transforms>

              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">

                <exc14n:InclusiveNamespaces PrefixList="wsu wsse S" />

              </ds:Transform>

            </ds:Transforms>

            <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <ds:DigestValue>r5dVsFsteThPivLkpsO+Fme5FIs=</ds:DigestValue>

          </ds:Reference>

        </ds:SignedInfo>

        <ds:SignatureValue>8PLYt+ddRqRV2i7tX2R7PgIEJU=</ds:SignatureValue>

        <ds:KeyInfo>

          <wsse:SecurityTokenReference wsu:Id="uuid_ba14a75f-0bee-4ec6-ae77-2646ab277e2f" wsse11:TokenType="http://docs.oasis-open.org/wss/oasis-wss-soap-message-security-1.1#EncryptedKey">

            <wsse:Reference URI="#_5002" ValueType="http://docs.oasis-open.org/wss/oasis-wss-soap-message-security-1.1#EncryptedKey" />

          </wsse:SecurityTokenReference>

        </ds:KeyInfo>

      </ds:Signature>

    </wsse:Security>

</S:Header>

<S:Body wsu:Id="_5007">

    <xenc:EncryptedData xmlns:ns19="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity" xmlns:ns18="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" xmlns:ns17="http://www.w3.org/2003/05/soap-envelope" Id="_5008" Type="http://www.w3.org/2001/04/xmlenc#Content">

      <xenc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc" />

      <ds:KeyInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="KeyInfoType">

        <wsse:SecurityTokenReference wsse11:TokenType="http://docs.oasis-open.org/wss/oasis-wss-soap-message-security-1.1#EncryptedKey">

          <wsse:Reference URI="#_5002" ValueType="http://docs.oasis-open.org/wss/oasis-wss-soap-message-security-1.1#EncryptedKey" />

        </wsse:SecurityTokenReference>

      </ds:KeyInfo>

      <xenc:CipherData>

        <xenc:CipherValue>+VCg/Fg9BR...</xenc:CipherValue>

      </xenc:CipherData>

    </xenc:EncryptedData>

</S:Body>
</S:Envelope>

This looks a little intimidating, but I was actually able to create a Wcf custom binding which generates this exact message:

private static CustomBinding CreateCustomBinding(EndpointAddress address)
{

            var res = new CustomBinding();

            SymmetricSecurityBindingElement sec =
                (SymmetricSecurityBindingElement)SecurityBindingElement.CreateMutualCertificateBindingElement(

                    MessageSecurityVersion.
     WSSecurity11WSTrust13WSSecureConversation13WSSecurityPolicy12BasicSecurityProfile10);

            res.Elements.Add(sec);

            sec.SetKeyDerivation(false);

            sec.MessageProtectionOrder = MessageProtectionOrder.SignBeforeEncrypt;

            sec.DefaultAlgorithmSuite = SecurityAlgorithmSuite.Basic128;

            sec.EndpointSupportingTokenParameters.Signed.Add(sec.EndpointSupportingTokenParameters.Endorsing[0]);

            sec.EndpointSupportingTokenParameters.Endorsing.Clear();

            sec.EnableUnsecuredResponse = true;

            res.Elements.Add(new CustomTextMessageBindingElement());

            res.Elements.Add(new HttpTransportBindingElement());

            return res; }

At this point everything was smelling like roses – I was able to get a real response from the server instead of the error above.

However from a reason unknown to me at the time the Wcf client was rejecting the response with this error:

Message security verification failed.

"End element 'Reference' from namespace 'http://www.w3.org/2000/09/xmldsig#' expected. Found element 'ds:DigestMethod' from namespace 'http://www.w3.org/2000/09/xmldsig#'. Line 1, position 2075."

Here is the relevant response section:

<ds:Signature Id="_1">

    <ds:SignedInfo>

       ...

        <ds:Reference URI="_5002">

           <ds:Transforms />

           <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

           <ds:DigestValue>2jmj7l5rSw0yVb/vlWAYkK/YBwk=</ds:DigestValue>

        </ds:Reference>

       ...

    <ds:SignedInfo>
</ds:Signature>

This is the response signature, which seems pretty straight forward. Why would Wcf throw an exception?

Analysis

If you take a closer look at the response above, you can see how it contains an empty “transforms” element. A non empty one looks like this:

          <Reference URI="#_0">

            <Transforms>

              <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />

            </Transforms>

            <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

            <DigestValue>jcLdOx6w1477YMoCJR8TJQRJuKw=</DigestValue>

          </Reference>

This is the framework code that parses the signature transforms element:

        public void ReadFrom(XmlDictionaryReader reader, TransformFactory transformFactory, DictionaryManager dictionaryManager)

        {

            reader.MoveToStartElement(dictionaryManager.XmlSignatureDictionary.Transforms, dictionaryManager.XmlSignatureDictionary.Namespace);

            reader.Read();

            while (reader.IsStartElement(dictionaryManager.XmlSignatureDictionary.Transform, dictionaryManager.XmlSignatureDictionary.Namespace))
            {

                //handle transformation
            }
            reader.MoveToContent();

            reader.ReadEndElement(); // Transforms             ...
        }

There are 3 cases that this code is supposed to handle:

In case the transforms element has childs (as in the last snippet before the above code), then the Read() call would put the cursor on the first child element (“transform” in singular). The while loop will end at the closing “transforms” element which is aligned with the following ReadEndElement() call.

In case there are no childs but there is a separate closing element:

          <Reference URI="#_0">

            <Transforms>

            </Transforms>

            <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

          </Reference>

then Read() would put us on the closing element, and since we will not enter the loop ReadEndElement will be right again.

In case of an empty element:

<Reference URI="#_0">

<Transforms />

<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />

</Reference>

the Read() would advance us behind the transformations tag! The next tag is the opening DigestMethod which of course fails for ReadEndElement().

So the case of an empty transforms element is responsible for the interoperability bug.

Workaround

If you are in a position to change the output of the Java server then various workarounds may apply. But what if you can’t?

The only real solution seems to be implementing the relevant ws-security portion by yourself. Here I show how to do the bare minimum of such a scheme – use the wcf security stack for the sending of the message and decrypting the response by ourselves. The missing piece will be the signature validation, which is very important in production but we can continue with development even without it as at least we get a readable response.

My solution involves using a custom message encoder. I have used the Wcf SDK message encoder sample as a template. When the response comes back to the encoder it decrypts it and remove the security from it before passing it to the upstream channel. To make sure the security channel will not fail (due to the now unsecured response) we verify our binding has EnableUnsecuredResponse set to true.

Here is how the encoder code looks like:

        public override Message ReadMessage(Stream stream, int maxSizeOfHeaders, string contentType)

        {
            var sr = new StreamReader(stream);

            var wireResponse = sr.ReadToEnd();

            var logicalResponse = GetDecryptedResponse(wireResponse);

            logicalResponse = String.Format(

                    @"<s:Envelope xmlns:s=""http://schemas.xmlsoap.org/soap/envelope/"">

                            <s:Body>

                                {0}

                            </s:Body>

                        </s:Envelope>",

                    logicalResponse);

            XmlReader reader = XmlReader.Create(new StringReader(logicalResponse));

            return Message.CreateMessage(reader, maxSizeOfHeaders, MessageVersion.Soap11);
        }

        private string GetDecryptedResponse(string encryptedResponse)
        {
            var doc = new XmlDocument();

            doc.LoadXml(encryptedResponse);

            var cipherNode = doc.SelectSingleNode("//*[local-name(.)='Body']//*[local-name(.)='CipherValue']");

            byte[] cypher = Convert.FromBase64String(cipherNode.InnerText);

            byte[] key = GetEncryptingKey();

            byte[] iv = GetIV(cypher);

            var AesManagedAlg = new AesCryptoServiceProvider();

            AesManagedAlg.KeySize = 128;

            AesManagedAlg.Key = key;

            AesManagedAlg.IV = iv;

            var body = ExtractIVAndDecrypt(AesManagedAlg, cypher, 0, cypher.Length);

            return UTF8Encoding.UTF8.GetString(body);
        }

So we find the ciphered portion of the message and decrypt it. We then use it to construct a new unsecured message which we will pass along in the pipeline. Here this is done in a dirty way which hard codes the soap11 envelope.

But where do we get the security key to decrypt the message from?

This key is encrypted in our request:

      <e:EncryptedKey Id="uuid-7674d043-9a35-495d-a4dc-27daee955237-1" xmlns:e="http://www.w3.org/2001/04/xmlenc#">

           …

        <e:CipherData>

          <e:CipherValue>HQ0UzEZnc9lq...</e:CipherValue>

        </e:CipherData>

</e:EncryptedKey>

We cannot decrypt it since it is encrypted with the server public key and we do not have the private one. However the security channel must keep reference to this key somewhere so that it will be able to decrypt the response (which it would do anyway unless the empty transformation bug). We need to use some reflection tricks to get the session key:

        private void SaveEncryptionKey(Message message)
        {
            var secHeaderType = message.GetType().GetField("securityHeader",
                                              BindingFlags.NonPublic | BindingFlags.Instance);

            var secHeader = secHeaderType.GetValue(message);

            var encTokenType = secHeader.GetType().BaseType.BaseType.GetField("encryptingToken",
                                              BindingFlags.NonPublic | BindingFlags.Instance);

            var token = (WrappedKeySecurityToken)encTokenType.GetValue(secHeader);

            var securityKey = (InMemorySymmetricSecurityKey)token.SecurityKeys[0];

            var symmetricKey = securityKey.GetType().GetField("symmetricKey",
                                               BindingFlags.NonPublic | BindingFlags.Instance);

            this.key = (byte[])symmetricKey.GetValue(securityKey);
        }

The full source code is here. It will not run since for privacy reasons I did not use the real wsdl / certificates. But it can be used as a reference in case you encounter a similar situation.

Wcf trick: Dynamically change proxy address

2010-12-17T22:00:00.000+02:00

It is quite common to have a few service environments, for example one for testing and one for production. One way to switch a Wcf client from one environment to another is by changing the address in app.config:

<endpoint address="http://localhost/Service.svc" ... />

However sometime we need to dynamically change the address from code due to some logic.
The naive approach would be to do something like this:

MyService client = new MyService("WSHttpBinding_MyService", 

        "http://new_server/Service.svc")

The reason this is naive is that app.config may contain additional information on the endpoint, namely its identity and headers:

<endpoint address="http://localhost/Service.svc" Name="WSHttpBinding_MyService" ...>

      <identity>

         <dns value="ServerIdentity"/>

      </identity>

   </endpoint>

when we create the proxy with a different endpoint in the constructor we override the identity information. This may result in this error:

Identity check failed for outgoing message. The expected DNS identity of the remote endpoint was 'localhost' but the remote endpoint provided DNS claim 'ServerIdentity'. If this is a legitimate remote endpoint, you can fix the problem by explicitly specifying DNS identity 'ServerIdentity' as the Identity property of EndpointAddress when creating channel proxy

The solution
Create the proxy normally. Then separately assign the new endpoint address keeping the identity and header values:

proxy.Endpoint.Address = new EndpointAddress(

    new Uri("http://new_server/Service.svc"), 

    proxy.Endpoint.Address.Identity, 

    proxy.Endpoint.Address.Headers);

Wcf and WebLogic security interoperability

2010-12-12T20:35:00.000+02:00

Yesterday I have mentioned some recent white papers about web services interoperability.
Today I've stumbled in another recent article which shows username over certificate interoperability between Oracle WebLogic Server 11g and Microsoft.NET WCF 4.0. This article has step by step instructions to make it happen.

Web services interoperability white papers

2010-12-11T22:30:00.000+02:00

A few years ago a white paper on web services interoperability would start with "turn off secure conversation" or "avoid the following types". A recent set of white papers published by Microsoft shows the today all vendors are much more mature and ready for real world interoperability.

Abu Obeida and Mike Champion from Microsoft announced recently the below articles. I recommend this reading for those of you who need to write interoperable web services. Of course not each article is relevant for each interop scenario so guess what? I've read them for you so you can decide which ones are relevant for you.

Data types interoperability between .NET and Java
Christian Weyer and Buddhike de Silva have listed most types you might ever want to use with web services and verified their interoperability status. This is a must read for every public facing web service developer.
One thing that I am missing here though is the Axis2 java stack which is very popular. Also I'm confused by the check on WebSphere - as an application server you can use various stacks on it so I'm not sure which one was used. I assume Wcf was used on the .Net side (".Net 4" is specified) and I'm very curious if the data contract was able to cope with all scenarios or was xml serialization required in some cases. Generally data contract are optimized to enforce "best practice" which is sometimes counterproductive for interoperability.

The conclusion is very promising - good interoperability exists in almost all scenarios. One important reminder is the lack of interoperability with Hashtable and some of the generic types.

Security Interoperability Guideline between WCF-WIF and Oracle WebLogic

Jesus Rodriguez wrote a great article on ws-security interop between Wcf and Oracle WebLogic. Naturally the article dives into the ws-security details and is very technical. It proves that X.509 message security is a good option for interoperability. While jesus shows how to achieve federation interoperability via SAML and WS-Trust, it is clear that there is still a way to go here as a holder-of-key scenario is still not possible. I have not tested it myself but hopefully bearer tokens should also work well with these parties. The article does not mention it as it deals with message security while bearer tokens are a mix of message and transport.

Security Interoperability Guideline between WCF-WIF and IBM WebSphere Application Server

Jesus adds another good one and this time on interoperability between Wcf and WebSphere. The username and x.509 scenarios work pretty well. The article mentions that WS-Trust interoperability is possible, but does not give the details, so I'm not sure if it is a mistake in the upload or if it really required some tweaks which were out of scope. Either way this is yet another proof that there is still a way to go with federation interoperability. Such security scheme will be very popular in the cloud era so hopefully all vendors will keep investing in it. As it stands now Wcf+Wif have great support for the related scenarios so I'll report on any news from the other vendors.

As a side note, While MTOM interoperability is possible here, I have seen in the past a few gotchas so hopefully a future paper will cover that up (I'll try to write about it myself actually).

Standards-Based Interoperability between SAP NetWeaver and Microsoft .NET Framework
A rich set of scenarios are working in this case, including security and MTOM. It is a little disappointing that a basic scenario like custom headers only partially pass, but NetWeaver is an opinionated framework that expose its users only what is relevant for them.

Metro to WCF Interoperability
The combination of Metro and Wcf shows off the richest set of working interoperability scenarios. This is not surprising considering the fact that one of the goals of WSIT (Metro's security stack) is interoperability with Wcf.

To conclude, all articles are very interesting and show that interoperability is a reality.

Silverlight 5: More details on WS-Trust support

2010-12-04T20:28:00.000+02:00

Yesterday Microsoft had announced Silverlight 5 Beta to be released in the first half of 2011. Sure, there are plenty of great features there, but if you are readers of my blog I'm sure you are especially interested in a particular one. Here is a hint (click to enlarge):

Yes, more of the WS-* stuff is making its way into SL.

While this is all good news, the announcement is missing more details as for what exactly will be supported. This post will analyze the status of SL & WS-Trust in SL 4 + following unofficial projects, and based on the missing gaps I will try to estimate what the upcoming release *might* include.

Background - Federation and WS-Trust
This post is not aimed to be an introduction to federation or to claims based programming, but as a quick reminder here is the general flow:

Our environment contains:

Application Service. This service contains some operations that clients want to consume. The service will only accept client requests that embed a token gotten from the STS (usually a Saml token, which is an Xml token format). One reason might be that the service does not want to deal with authentication and leaves this mission to the STS which is developed by a dedicated team.

STS. STS role is to authenticate clients and generate tokens that the client can present to the service. This is a little oversimplified, mainly since there are some STS that only aim to provide client identity, and some that provide protection over the resource (service), and in many cases both of them will be used.

Client. The client is any application (client or server) that wants to consume the application service and needs to get a token from the STS first. The client is of course not necessarily using Silverlight.

Needless to say is that Wcf is a great framework for such scenarios.

How Silverlight fits this picture?

A lot of organizations already have security infrastructure that contains STS. They would like SL applications to work with it also, mainly so SL applications can call other web services which expects a Saml identity.

What is the current status of SL & WS-Trust (SL4+ unofficial projects)

Until the beginning of 2010 there was no built-in support for such scenarios, and some creative solutions were used. Earlier this year Caleb Baker had disclosed a project that brings WIF support to SL as part of the identity developer training kit. WIF is (among other stuff) a set of extensions to WCF that makes working with claims easier. Since Saml is very aligned with the claims model, and STS usualy emits Saml over WS-Trust protocol, this effectively brought WS-Trust support for SL for the first time.

What *might* SL5 support?
In other words we need to ask what is missing after the mentioned WIF support.

As far as I know the mentioned support was not official but a part of a training kit. Many organization will not use it until it is official.

WIF generally work on XP only. I say "generally" because the WIF/SL seems to be standalone so I'm not sure if that's the case with it also. Anyway since SL is not windows-only we will probably get this subset of WIF on all platforms.

WIF works well, but it is a different programming model than the Wcf WSFederation binding. Possibly Microsoft will consider supporting the native Wcf model as well.

STS is not limited to bearer tokens as the current WIF implementation is. This is actually an SL limitation since it only supports a subset of the Wcf bindings and most of the X.509 certificate settings are not supported. Microsoft might consider to support more bindings which might be used to communicate with the STS / application service.

The current training kit does not natively support multi leg federation (e.g. more than one STS), although dominick had published the a way to fix it.

Again, these are just my guesses / wish-list, but either way this is great news for every silverlight developer.

WsiHero: A GUI utility for web services WSI validation

2010-11-20T14:10:00.001+02:00

Download WsiHero (3MB)

Last week was an important moment for web services interoperability: WS-I has approved the final versions of Basic Profile (BP) 1.0 and 2.0 and of Reliable Secure Profile (RSP) 1.0. These profiles are a set of rules which web services vendors and consumers are expected to comply with in order to support interoperability. The original profile goes back to 2004 and was highly successful in improving web services interoperability.

So how do we test our web services for WSI profiles compliance?
The WSI site contains a deliverables section which has a few command line utilities to check the services. Using these command lines directly has some overhead like a learning curve for all the options, preparing the input wsdl / soap files in the specific log format, and repeating this manual task every time.

I have written the WsiHero utility to make this validation simpler.

How to become a WSI hero?
If you want to easily validate your service for WSI compliance and become a WSI hero then Download WsiHero. It is a GUI application which saves you the need to deal with the WSI command line and its strict log format.

Installation
1. Ensure Microsoft .Net framework 2.0 or higher is installed (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=0856eacb-4362-4b0d-8edd-aab15c5e04f5&displaylang=en).

2. Ensure Java 5 or higher is installed (http://www.java.com/en/download/index.jsp).

3. Download WsiHero and extract it to some folder, for example c:\program files\WsiHero.

4. Since it is not legal for me to distribute the WSI command line utility, please download it by yourself as follows:

Download these two zip files to your machine:

http://ws-i.org/profiles/BPTestToolsProcess-1.2-2.0-Final.zip
http://www.ws-i.org/profiles/RSP1.0-Delivery-Package.zip

Do not extract them, just put them as .zip in the WsiHero folder (c:\program files\WsiHero). Do not change their names.

In case the direct links do not work for some reason here is how to get the above files from the WSI site: From the "deliverables" section of the site download "BP 1.2 and 2.0 Test Tools package" and "Reliable Secure Profile Delivery Package".

Usage
1. Run WsiHero.exe from its folder.

2. Paste a Wsdl or a Soap file location into the text box and press "Add".

Notes:

Location can be a local disk path or a Url.

If the file contains Soap it must have the .Xml extension. Otherwise it is assumed to be a Wsdl.

For example any of the following can be put in the text box:

c:\soap_envelopes\soap.xml

c:\wsdl\service.wsdl

http://www.service.com/MyService.svc?wsdl

If you do not have any Wsdl available you can to use http://webservices.daehosting.com/services/TemperatureConversions.wso?WSDL

3. Repeat step (2) any number of times.

4. Click the arrow near the validation button and choose the validation type you want to perform:

For Soap 1.1 services choose "Basic Profile 1.2".

For Soap 1.2 services choose "Basic Profile 2.0".

For secured / reliable services choose "Reliable Secure Profile 1.0" (you may want to test such services also with one of the former options).

After a few seconds (depending on the input size) the WSI report will appear:

I plan to open source and enhance this utility soon.
Let me know if you have any problem or a feedback.

Wcf with 256 bit Ssl

2010-11-08T23:24:00.000+02:00

Recently I had to call a Java web service from Wcf. The service was secured with a 256-bit ssl certificate. Every request I made was rejected with a 401 Unauthorized response.

I then used this site to check what encryption strength my machine supports:

It seemed my pc is configured to only allow 128 bit connection.

Luckily I have found this link which explains how to configure windows to use 256-bit Ssl (Vista only). It seems the problem is in the windows defaults and not in Wcf. This changed the result in IE:

and also the Wcf client worked. The strange is that the setting that needs to be changed should only affect the order of the supported algorithms. So if the server requires a 256-bit OR a 128-bit key, the default setting would suggest 128 first. But if the server only allows 256-bit I would expect this to work since this is a supported configuration, just not the first in order. Either way the fix made it work.

Silverlight, Html5 and the future of Wcf

2010-11-01T00:09:00.006+02:00

Update: Silverlight is alive and well! Microsoft clarifies that Silverlight is a strategic technology. Scott Guthrie promises that future features and tools will be revealed soon. I will update about anything which has relevancy to Wcf. This post is still very relevant to show the increasing importance of javascript clients, side-by-side with Silverlight clients.

Everybody talk about the "unofficial" announcement of Microsoft embracing Html5 over Silverlight. And while this was never approved (unless you read between the lines) the below may be a hint.

Yesterday Microsoft had published an "early access" codeplex project with some features from the next Wcf version. The two main themes seem to be native Http support and jQuery. And not a word about Silverlight. Will there ever be security in the SL netTcp transport? Will we see more WS-Security coverage from the Wcf stack? It seems jQuery improvements got a higher priority for this release. Is the Wcf early access project a hint for a focus shift from SL to Html5? (update: more SL features will be revealed soon, hopefully they include the above).

The planned jQuery support includes code generation for javascript clients, working with untyped json objects, Html forms submission and more.

So how does the Html5 thing affect Wcf?
This is definitely a big push to the Rest / Json / Anti WS-* approach (not that it needed it anyway). Silverlight did not support the full Wcf stack in version 4, but in theory nothing prevented it from growing into it in the next versions. Browsers in contrary cannot work with WS-Security and feel much more at home with Json over Xml. Also Rest is more natural for the web anyway.

Important Wcf performance issue + workaround

2010-10-27T22:44:00.007+02:00

I have written about Wcf performance issues before, but this one seems to be the biggest. Valery had published in the Wcf forum an interesting performance issue. In short, a WCF client tries to consume a non-WCF service where the contract looks something like this:

class Foo

{

   byte[] picture;

}

In soap, byte arrays are encoded as base64 strings so it can look like this:

<picture>/9j/4AAQSkZJReV6R8MLi7nW6UUUViWf/Z.....</picture>

or with line breaks after each 73 characters, like this:

<picture>/9j/4AAQSkZJReV6R8MLi7nW61+58zBz5Q+7Xpdj

/PK/4AAQSkPOIeV6R8MLi7nW61+58zBz5Q+7Xpdj

/9R/4AAQSkZJReV6R8MLi7nW6VZ788zBz5Q+7Xpdj

4U4wVoqwUUUViWf/Z</picture>

both options are valid according to the base64 RFC:

Implementations MUST NOT add line feeds to base-encoded data unless

the specification referring to this document explicitly directs base

encoders to add line feeds after a specific number of characters.

Ok so it does not really advocate this... But it is a fact that many soap stacks still use this MIME-originated format and also Wcf supports it.

So what is the problem?
It seems that when Wcf gets a message which contains base64 with CRLF, the processing is slower in a few seconds(!). A drill down shows that the problem is in the DataContract serializer. Take a look at this program:

[DataContract]

public class Foo

{

    [DataMember]

    public byte[] picture;

}

class Program

{

        static void Main(string[] args)

        {

            var t1 = getTime(@"C:\temp\base64_with_line_breaks.txt");

            var t2 = getTime(@"C:\temp\base64_without_line_breaks.txt");             

            Console.WriteLine("Time with breaks: " + t1);

            Console.WriteLine("Time with no breaks: " + t2);

            Console.ReadKey();

        }

        static double getTime(string path)

        {

            var ser = new DataContractSerializer(typeof (Foo));

            var stream = new FileStream(path, FileMode.Open);

            var start = DateTime.Now;

            for (int i = 0; i < 40; i++)

            {

                ser.ReadObject(stream);                 

                stream.Position = 0;

            }

            var end = DateTime.Now;

            var t = end - start;

            return t.TotalSeconds;

        }

}

For those of you who are interested to test this, the files are here and here.

The output is:

Time with breaks: 10.8998196 seconds

Time with no breaks: 0.0029994 seconds

This clearly reveals a performance problem.

Why does this happen?

While debugging the .Net source code, I have found this in the XmlBaseReader class (code comments were in the source - they are not mine):

int ReadBytes(...)

{

  try

 {

   ...

 }

   catch (FormatException exception)

  {

      // Something was wrong with the format, see if we can strip the spaces

      int i = 0;

      int j = 0;

      while (true)

      {

          while (j < charCount && XmlConverter.IsWhitespace(chars[j]))

              j++;

          if (j == charCount)

               break;

          chars[i++] = chars[j++];

      }

...

}

}

So the data contract serializer tries to read the base64 string, but for some reason succeeds only if the string does not have white spaces inside it (we can further debug to see how that happens but it is exhausting for one post :). The serializer then removes all the white spaces (which requires copying the buffer again) and tries again. This is definitely a performance issue.

Notes:

This happens with both .Net framework 3.5 and 4.0.

This is a DataContract specific issue - it does not happen when you use other .Net mechanisms such as Convert.FromBase64String

I have reported this in Microsoft connect, you are welcome to vote this issue up.

Workarounds

There a few workarounds. The trade-offs are generally convenience of API (or "where you prefer to put the 'dirty' stuff").

1. As Valery noticed, you can change the contract to use String instead of byte[]. Then Convert.FromBase64String will give you the byte array.

2. Change your contracts to use the XmlSerializer instead of DataContract serializer. The former does not experience this issue. The XmlSerializer is generally slower (when base64 does not appear that it) so this is what you loose. You get a better API here as clients do not need to manipulate the base64 string.

3. The best of course is to change the service implementation to return base64 without line breaks. Also if large binaries are returned anyway it may be a better idea to employ MTOM.

4. A Wcf custom encoder can strip the spaces from the message before it is deserialized. However this also involves copy of strings and this is beneficial only in rare cases.