Vital's Blog

Transfer large messages in WCF - Part 3

2011-07-06T05:18:00.000-07:00

This in continuation with Large Message Transfer in WCF Part 1 and Part 2.

In last article by using Buffered transfer mode and MTOM, we upload a file of 88 mb, we observed memory usage by the application is 400 MB. We will try to transfer same size file by setting transfer mode as Streamed

</binding>

</basicHttpBinding>

Let us upload file of 88 MB size and observe the memory consumption. Below is screen shot of task manager
At any point of time client used max memory of 5 MB where as in buffered mode it was 400 MB

400 MB and 5 MB numbers may vary depend on so many other factors, but my intention here is to show you the memory consumption difference between buffered and streamed transfer mode

Download this code and toggle transfer mode between buffered and streamed and observe memory consumption

Let’s go into some more details on streaming

MTOM helps with transfer overhead and processing performance, but still entire message is still loaded into memory. i.e when we upload a file of size 88MB client will load entire message into memory. Once file is loaded completely service start taking it.
In Streaming as soon as message starts loading into memory, service starts receiving
Streaming will lose fewer features like reliable messaging and message security
Streaming is supported by NetNamedPipeBinding, NetTcpBinding and Basic HttpBinding
Streaming will expose message body as stream, but header will still buffered
Following are the different options in Transfer Mode

Buffered – This is the default setting
Streamed- Streams both request and response
StreamedRequest – Request is streamed and response is buffered
StreamedResponse– Request is buffered and response is streamed

Points to note on when to close stream

WCF will close the stream after the final read

For Services returning streams
For clients sending streams

We must close the stream after the final read

For clients reading a returned Stream
For services reading an incoming stream

Hope this helps
Vital.

Transfer large messages in WCF - Part 2

2011-05-18T05:21:00.000-07:00

This is in continuation with Part 1. let us look into details of Message Transmission Optimization Mechanism (MTOM)

Enable MTOM for above service by setting messageEncoding = “Mtom”

</basicHttpBinding>

</bindings>

When we try to upload our file the bytes are converted into Base 64 encoded data in SOAP envelope.

By enabling MTOM, SOAP messages are sent as Mulitpurpose Internet Mail Extension (MIME) multipart/related content.

Here, data will not be encoded any more thus reducing processing overhead of encoding and decoding into Base 64. Another disadvantage with base 64 encoded data is that it increases the data size by approximately 33%. So by enabling MTOM we will gain this 33% size as well.

You can visualize the difference using fiddler.
Before

After

MTOM helps in improving transfer overhead and processing performance. But the entire message is still loaded into memory. Let us try to visualize this using windows task manager.

When I tried to upload a file of size 88 MB, client will load all Binary data(MTOM)/Base 64 data(Text) to memory. Once it completely loads data, the service will start receiving it and we can observe memory consumption in task manager. The client continuously increases memory consumption for our scenario till it goes up to 450 MB and then the memory consumption in service starts increasing. See below screen shot

We will see how we can improve this in the next part

Hope this helps
Vital

Transfer large messages in WCF - Part 1

2011-04-29T13:35:00.000-07:00

Pre Requisites

Readers must have knowledge in creating basic WCF service and must know how to host and consume a service.

Default max received message size in WCF is 65kb, so when we try to send messages more than 65kb, WCF will give errors. Let us create a basic service which will pass a message more than 65 k and see what happens and figure out what are different ways to handle large messages in WCF.

1. Create a service contract that takes byte[] as input as shown below:

[ServiceContract]

public interface IService1

{

[OperationContract]

void UploadFile(byte[] data);

}

2. Implement Iservice1 which will write byte[] into file on a server

public class Service1 : IService1

{

public void UploadFile(byte[] data)

{

Random x = new Random();

string fileName = string.Format(@"Test{0}.mp3", x.Next().ToString());

File.WriteAllBytes(fileName,data);

}

3. The app.config file for the hosting environment is given below:

<system.serviceModel>

<host>

</baseAddresses>

</host>

</endpoint>

</service>

</services>

</behavior>

</serviceBehaviors>

</behaviors>

</system.serviceModel>

4. Now consume this service in a client. Here we will convert a mp3 file into byte[] which is more than 65 kb and send it to service

FileStream fs = new FileStream(@"Test.mp3", FileMode.Open, FileAccess.Read);

byte[] data = new byte[fs.Length];

fs.Read(data,0,System.Convert.ToInt32(fs.Length));

ServiceReference1.Service1Client proxy = new ServiceReference1.Service1Client();

proxy.UploadFile(data);

5. Make sure you enable the trace at service so that we can see the detailed error messages

6. Make sure service is running and execute the client .

7. As expected we got an error (look into trace at service) which says “The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.”. So we have to increase MaxReceivedMessageSize as below

</basicHttpBinding>

</bindings>

a. You can set maxReceivedMessageSize upto 2GB (2147483647, i.e. Int32.MaxValue)

b. Interesting point over here is when we get above error do we need to increase maxReceivedMessageSize at service or client?

i. If we change maxReceivedMessageSize value at a service, it refers to the maximum message size that the service can receive. We need to change this value when we get large data from client

ii. If we change maxReceivedMessageSize value at client, it refers to the maximum message size that the client can receive. We need to change this value when we get large data from service

Since the data we are sending from the client exceeds the maximum permissible limit on the service, it is clear that for our problem we need to increase the maximum message size at service

c. maxReceivedMessageSize is a local behavior setting i.e if you do some changes to maxReceivedMessageSize value at service and try to refresh service reference at client this value will not be updated, it will be always default value(65536) at client. This make sense with explanation in point b that explains different meanings for this property at client and service .

We have seen how we can transfer large data in WCF. But when we are working with large data, the first things that come into our mind are memory and processing overhead. These things can be addressed using

Message Transmission Optimization Mechanism (MTOM)

Streaming

Let us go into details of each one with examples in Part 2 and Part 3.

Hope this helps
Vital

Implement Publisher Subscriber model in WCF

2011-03-02T10:30:00.000-08:00

Download Source Code here

Pre Requisites

Readers must be having knowledge in creating basic WCF service and must know how to consume it.

Here I tried to illustrate how we can achieve publisher subscriber model in WCF

To explain this Let me take one common real world scenario Stock Exchange, Where investors register to Service and service will notify to all investors when stock got changed

My intention here is to make you understand how we can achieve publisher subscriber model in WCF not to solve above mentioned problem.

Duplex Operations

To achieve this we will use duplex operations.

Duplex operations allows client to send a request to service which contains information about endpoint at client which will be used to get communication from service

Service must be hosted with a binding that supports duplex communication. Following are the bindings that support duplex communication

· Wsdualhttpbinding

· Tcpbinding

· Namedpipes

Let us start implementing duplex operations so that you will get more clarity on what I’m saying

To achieve our stock exchange scenario we need Service, Investor and Stock Exchange

Service

RegisterInvestorService: This service allows investors to register or de register to get updated when the stock price got changed

StockExchangeService: This service will be used by stock exchange to update price. When price got updated it will inform all the investors registered to the service

Investor

This consumes RegisterInvestorService and Registeritself to service and will get notifications from RegisterInvestorService

Stock Exchange

This consumes StockExchangeService and updates the price.

Service Implementation

Create Service Contracts as mentioned below

[ServiceContract(CallbackContract = typeof(IRegisterInvestorServiceCallback))]

public interface IRegisterInvestorService

{

[OperationContract(IsOneWay = true)]

void RegisterInvestor(Guid investorid);

[OperationContract(IsOneWay = true)]

void DeRegisterInvestor(Guid investorid);

}

public interface IRegisterInvestorServiceCallback

{

[OperationContract(IsOneWay = true)]

void StockUpdated(double price);

}

[ServiceContract()]

public interface IStockExcahangeService

{

[OperationContract(IsOneWay = true)]

void UpdateStock(double price);

}

Let me Explain Purpose of each interface

IRegisterInvestorService: This will be useful for the investors who want to register or de register to get updated when the stock price got changed

INotifyInvestors: This will be the contract which will used to notify the clients when stock got updated. Client will implement this and take action when stock price got changed

IStockExcahangeService: This will be useful for stock exchange to inform service when price got changed. Then this service will notify all its clients

When you observe above code you will find 2 peculiar things

1. CallbackContract: says who is the callback contract for this service contract. So that this service contract will notify all clients who registered with service using this call back contract

2. interface INotifyInvestors contains some Operation Contract but not having ServiceContract: This is called callback contract which will be implemented by the client (investors) and called by the service

Create below class in Service where it will store registered Investors information. This contains information about how we register and de register a investor and update all client who registerd with it when price got updated

public static class MicrosoftStock

{

private static Dictionary<Guid, INotifyInvestors> registeredInvestors =

new Dictionary<Guid, INotifyInvestors>();

public static void UpdateStock(double price)

{

foreach (KeyValuePair<Guid, INotifyInvestors> obj in registeredInvestors)

{

obj.Value.StockUpdated(price);

}

public static void RegisterInvestor(Guid investorid,

INotifyInvestors notifyInvestor)

{

registeredInvestors.Add(investorid, notifyInvestor);

}

public static void DegisterInvestor(Guid investorid)

{

registeredInvestors.Remove(investorid);

}

In above code you might have observed one thing that i kept MicrosoftStock as static so that all investors can share the information.

Implement IRegisterInvestorService

public class RegisterInvestorService : IRegisterInvestorService

{

public void RegisterInvestor(Guid investorid)

{

INotifyInvestors callback = OperationContext.Current.GetCallbackChannel<INotifyInvestors>();

MicrosoftStock.RegisterInvestor(investorid, callback);

}

public void DeRegisterInvestor(Guid id)

{

MicrosoftStock.DegisterInvestor(id);

}

Implement IStockExcahangeService

public class StockExchangeService : IStockExcahangeService

{

public void UpdateStock(double price)

{

MicrosoftStock.UpdateStock(price);

}

Below is the config for the service which is common for any other WCF service except we are using wsdualHttpBinding which supports duplex communication as I mentioned earlier.

<system.serviceModel>

</service>

</service>

</services>

</behavior>

</behavior>

</serviceBehaviors>

</behaviors>

</system.serviceModel>

Now the service is ready.. I leave it to you how you host the service

Investor Implementation

This is simple console application which consumes RegisterInvestorService(hope you are aware of how to consume WCF service)

First Step in Investor is implementing IRegisterInvestorServiceCallback, where we will get notifications. In implementation investor can take some action when price got changed

[CallbackBehavior(UseSynchronizationContext = false)]

class StockExcahageUpdates : IRegisterInvestorServiceCallback

{

public void StockUpdated(double price)

{

Console.WriteLine("Stock Updted: " + price.ToString());

}

Second step is register investor to the service and wait for notifications.

In this step you will observe one thing ServiceClient is accepting InstanceContext as parameter which is not required in traditional Request/Reply Service.

Using Instance context we will let service know to whom it needs to be notified when price got changed or we can say it as client’s end point to which service needs to be communicated

[CallbackBehavior(UseSynchronizationContext = false)]

class Program

{

static void Main(string[] args)

{

InstanceContext callbackInstance = new InstanceContext(new StockExcahageUpdates());

InvestorService.RegisterInvestorServiceClient proxy = new InvestorService.RegisterInvestorServiceClient(callbackInstance);

proxy.RegisterInvestor(Guid.NewGuid());

Console.WriteLine("Observing the Stock...");

Console.ReadKey();

}

Stock Exchange Implementation

This is simple Console Application which consumes StockExchangeService and update price to the service.

static void Main(string[] args)

{

StockExchangeService.StockExcahangeServiceClient proxy = new StockExchangeService.StockExcahangeServiceClient();

proxy.UpdateStock(1000);

}

Testing

1. Service must be in running state

2. Create couple investors by executing Invetors.exe couple of time. Then both investors will be registered to service and waiting for price change notification from service

3. Run Stock exchange which will update the price in service, then you can observe above 2 investors will get notified with updated price

Hope this helps

Vital.

How to Debug WCF

2011-02-28T23:34:00.000-08:00

I’m trying to share you a problem we got while consuming WCF Service and how did we solve
it. Which will help you debugging WCF.

While consuming WCF service we got error message “The remote server returned an
unexpected response: (400) Bad Request.”.

In that Message I didn’t understand what Bad means. It would be helpful if we are having more detailed information on what Bad means?

To know this we need to enable tracing at WCF Service end.

To enable tracing add below config entries in your service config file under configuration.

<system.diagnostics>
    <sources>
      <source name="System.ServiceModel" switchValue="Warning">
        <listeners>
          <add name="WcfLogSample" />
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add name="WcfLogSample" type="System.Diagnostics.XmlWriterTraceListener"
             initializeData="D:\log\Traces2.svclog" traceOutputOptions="Callstack" />
    </sharedListeners>
   <trace autoflush="true" />
</system.diagnostics>

Then try to reproduce the error from client again and open File "D:\log\Traces2.svclog" .where you can see detailed error message as below

“The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.”"

Then simply go ahead and increase value MaxReceivedMessageSize. Now the problem is solved

What I’m trying to say here is, make your life easy while debugging by enabling tracing
in WCF when we get weird error.

Hope this helps

Vital

How to host WCF Service in WAS

2011-02-24T05:29:00.000-08:00

Here I tried to illustrate how to host WCF Service in WAS with minimum required steps.

Prerequisites

Need Windows server 2008, IIS7 and WAS
Readers must be aware of WCF basics

Create Basic WCF Service

1. Create New Project, select WCF Service Application as Template with Name “SampleWCFHosting”

2. Replace below config entries in web.config file

<system.serviceModel>

<servicename="SampleWCFHosting.Service1"behaviorConfiguration="servicebehavior">

<endpointaddress="netPipe"contract="SampleWCFHosting.IService1"binding="netNamedPipeBinding"/>

<endpointaddress="netTcp"contract="SampleWCFHosting.IService1"binding="netTcpBinding"/>

<endpointaddress="basicHttp"contract="SampleWCFHosting.IService1"binding="basicHttpBinding"/>

<endpointcontract="IMetadataExchange"binding="mexHttpBinding"address="mex" />

</service>

</services>

<behaviorname="servicebehavior">

<serviceMetadatahttpGetEnabled="true"/>

<serviceDebugincludeExceptionDetailInFaults="false"/>

</behavior>

</serviceBehaviors>

</behaviors>

<serviceHostingEnvironmentmultipleSiteBindingsEnabled="true" />

</system.serviceModel>

3. Build Solution and publish it as shown below

Install Service in WAS
1. Open IIS Manager

2. Expand Sites, right click and click on “Edit Bindings” of Web site under which you want to create application, then add entries as below

3. Right click and click on “Add Application“ on above Web site and Alias and Physical path as below

4. Right Click “WCFOnWAS” Application and click on Manage Applicationà Advanced Settings and Update Enabled Protocols with “http, net.tcp, net.pipe”

Now you have completed hosting WCF on WAS

Just to see whether our Service is accessible browse http://localhost/WCFOnWAS/Service1.svc

Test Service from Client

Let us test it using some client application

1. Create Console Application

2. Add service Reference usinghttp://localhost/WCFOnWAS/Service1.svc

3. After adding service reference app.config is updated with different bindings available

4. Add below code in Main to test WCF

//get Binding Name from app.config and pass as parameter for Service1Client

//Change different Binding Names to test different Bindings

ServiceReference1.Service1Client proxy = new ServiceReference1.Service1Client("NetTcpBinding_IService1");

string s = proxy.GetData(1);

Get different binding names from app.config and pass as parameter for serviceclient to test different bindings.

Hope this helps

Vital