LEX's blog

Generics. Reference summary.

Tue, 31 Jan 2012 22:59:00 +0000

People say generic collections offer a lot of benefits compared to the non-generic ones. This post is a short reminder to myself (mostly) as of what are those benefits.

1 Memory usage

I start with a few lines of code in C#4, all pretty basic stuff. I ran it in Debug mode and waited till debugger stopped at the last line of main method. Then I got Immediate Window (in Visual Studio) opened, loaded SOS extension and dumped all the objects' sizes. Below is the code and in comments I put the extracted sizes of objects.

==Immediate Window==
.load sos
!ObjSize

==Code==
int count = 1000;
var arrayList = new ArrayList(count);        //4040 bytes
var byteList = new List<byte>(count);        //1036 bytes

var stack = new Stack(count);                //4040 bytes
var genericStack = new Stack<byte>(count);   //1036 bytes

var queue = new Queue(count);                //4052 bytes
var genericQueue = new Queue<byte>(count);   //1044 bytes

Apparently, this is all platform, CLR (pointers on 32-bit and 64-bit machines are different and different versions can use updated code), hardware etc etc dependent, but to me it's enough to see a generic collection use much less memory with value types. Overheat for storing boxed objects is quite large (nearly 4 times in this sample!).

2 Performance

I tested adding and taking 10.000.000 items into generic and non-generic collections. Here's a snippet of code and the results.

class Program
{
	static Random r = new Random();

	static void Main(string[] args)
	{
		//Init	
		int count = 10000000;
		
		byte[] values = new byte[count];
		r.NextBytes(values);
		
		var arrayList = new ArrayList(count);
		var byteList = new List<byte>(count);
		//...
			
		//Timed stress-load: add then take
		var arrayListTime = TimeAction(() =>
		{
			for (int i = 0; i < values.Length; i++)
			{
				var value = values[i];
				arrayList.Add(value);
			}
			for (int i = 0; i < values.Length; i++)
			{
				var value = values[i];
				int k = (byte)arrayList[i];
			}
		});
		//...
		
		//Output results
		Console.WriteLine("array list: " + arrayListTime);		
		//...
	}

	private static TimeSpan TimeAction(Action action)
	{
		var sw = new Stopwatch();
		sw.Start();
		action.Invoke();
		sw.Stop();
		return sw.Elapsed;
	}
}

And the output is

-----------------------------
-----------------------------
array list: 	2.2440829 sec
generic list: 	0.2694271 sec
-----------------------------
stack: 		2.0472536 sec
generic stack: 	0.3369304 sec
-----------------------------
queue: 		1.9683970 sec
generic queue: 	0.3751191 sec
-----------------------------
-----------------------------

It can be clearly seen that generics outperform non-generic collections. The time gain happens because generic collection doesn't spend time on boxing/unboxing and subsequent copy of the value to the value-type location.

3 Clean code

Consider a situation where one developer thinks he works with fruits, but somebody else has made an error and used an incorrect type.

stack.Push(new Apple());
stack.Push(new Dog());
var item = stack.Pop(); 
//you cannot peel a dog 
//but guess when you learn this?
((Fruit)item).Peel();

4 Variance

Generic classes are invariant in .NET2 - .NET4 inclusive. However in .NET 4 delegates and interface got support for one-directional variance (via either in or out keywords). Because it's a big topic I shall stop here and refer the reader to thoughts on stackoverflow and msdn.

5 Limitations

Generics support only classes, structs,interfaces, delegates and methods. It's not possible to parametrize events, constructors, properties, indexers etc.

Performance gain is not noticeable for reference type's collections (as boxing/unboxing/copying does occur) and for single parametrized item (as it's just too quick to make an effect).

If you enjoyed this post, make sure you subscribe to my RSS feed!

CLR Internals > AppDomain

Fri, 27 Jan 2012 13:39:00 +0000

Consider very simple code

1	class Program
2	{
3		static void Main(string[] args)
4		{
5			AppDomain newDomain = AppDomain
6				.CreateDomain("MyDomain_1");
7  		}
8	}

Question: How many domains are there at line 7, given this code run as a managed console application?

Think about it before continuing reading.

Without additional searching of the Internet or digging clever books it is easy enough to check what's happening. CLR has had several debuggers that allow developers to to debug the code on the edge of managed and unmanaged code. One of these debuggers is Son of Strike (SOS).

Let's get our hands dirty. I put a break point at line 7 and start debugging in Visual Studio. Once the breakpoint is hit, I load SOS from the Immediate Window in Visual Studio. There are usually some troubles with just loading the debugger extension, so consider this post to troubleshoot SOS.

.load sos

This will hopefully load SOS and now let's run the command to get all AppDomains:

!DumpDomain

The output on my machine was

--------------------------------------
System Domain:      59da1478
LowFrequencyHeap:   59da1784
HighFrequencyHeap: 59da17d0
StubHeap:           59da181c
Stage:              OPEN
Name:               None
--------------------------------------
Shared Domain:      59da1140
LowFrequencyHeap:   59da1784
HighFrequencyHeap: 59da17d0
StubHeap:           59da181c
Stage:              OPEN
Name:               None
Assembly:           00240a20 [C:\Windows\Microsoft.Net\assembly\GAC_32\...\mscorlib.dll]
ClassLoader:        00240ac0
Module Name
589e1000            C:\Windows\Microsoft.Net\assembly\GAC_32\...\mscorlib.dll

--------------------------------------
Domain 1:           001f19f0
LowFrequencyHeap:   001f1d6c
HighFrequencyHeap: 001f1db8
StubHeap:           001f1e04
Stage:              OPEN
SecurityDescriptor: 001f3168
Name:               Test.exe
Assembly:           00240a20 [C:\Windows\Microsoft.Net\assembly\GAC_32\...mscorlib.dll]
ClassLoader:        00240ac0
SecurityDescriptor: 0023b5a8
Module Name
589e1000            C:\Windows\Microsoft.Net\assembly\GAC_32\...mscorlib.dll

Assembly:           0024b048 [C:\Projects\Private\Test\bin\Debug\Test.exe]
ClassLoader:        0024b0e8
SecurityDescriptor: 0024c0e0
Module Name
003c2e9c            C:\Projects\Private\Test\bin\Debug\Test.exe

--------------------------------------
Domain 2:           0024f730
LowFrequencyHeap:   0024faac
HighFrequencyHeap: 0024faf8
StubHeap:           0024fb44
Stage:              OPEN
SecurityDescriptor: 00252c28
Name:               MyDomain_1
Assembly:           00240a20 [C:\Windows\Microsoft.Net\assembly\GAC_32\...mscorlib.dll]
ClassLoader:        00240ac0
SecurityDescriptor: 002542c8
Module Name
589e1000            C:\Windows\Microsoft.Net\assembly\GAC_32\...mscorlib.dll

Answer: so as you can see 4 domains will get created. Namely: system, shared, one default user domain and the last one that the user code additionally creates (lines 5-6). I found really nice article describing what happens when CLR starts and how CLR bootstrapper loads domains.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Basic implementation of StringBuilder and why it is required

Sun, 01 Jan 2012 16:51:00 +0000

This is one of the typical interview questions: when do I need to use a StringBuilder and how a simple implementation would look like?

The problem with a System.String type is that it is actually an immutable reference type although it seems to behave like mutable value type. Immutability means that for any write operation performed with the same string, a new object is mandatory created. For example, s1 = s1 + s2 will not modify s1, but will create a new object with a concatenated s1 and s2 and drops the reference to the old s1 to garbage collector. This is required to support long strings (< 2GB in .NET 4), save time and space complexity and maybe (which would be a pure guess) it was easier to follow a successful example of Java strings.

StringBuilder class has internal access to the string object and is useful for any string manipulations, especially numerous. Internally, StringBuilder uses string object, and since .NET 4 it uses a char array. This class does not create copies of string objects but rather acts as if it works with one mutable string

Because StringBuilder implements a Builder design pattern any state-changing operation should return the object itself (this). ToString() method acts as a Build() method and as it is defined in the System.Object, there is no need to include ToString() in the contract.

public interface ISimpleStringBuilder
{
	ISimpleStringBuilder Append(string value);
	ISimpleStringBuilder Clear();
	int Lenght { get; }
	int Capacity { get; }
}

A very simple implementation of the builder class may look like this

public class SimpleStringBuilder : ISimpleStringBuilder
{
    private char[] _internalBuffer;

    public ISimpleStringBuilder Append(string value)
    {
        char[] data = value.ToCharArray();

        //check if space is available for additional data
        InternalEnsureCapacity(data.Length);

        foreach (char t in data)
        {
            _internalBuffer[Lenght] = t;
            Lenght++;
        }

        return this;
    }

    public override string ToString()
    {
        //use only non-null ('\0') characters
        var tmp = new char[Lenght];
        for (int i = 0; i < Lenght; i++)
        {
            tmp[i] = _internalBuffer[i];
        }
        return new string(tmp);
    }
    ...    
}

This code is of course very basic, inefficient, not thread-safe, doesn't make any input validation etc. It does however demonstrates the idea behind StringBuilder class.

SimpleStringBuilder.zip (75.77 kb) [Downloads: 14]

If you enjoyed this post, make sure you subscribe to my RSS feed!

Sample WCF contract callback

Sat, 24 Sep 2011 14:41:00 +0000

I was trying to create a simple project based on WCF where a service can notify clients about something (as oposed to the standard scenarios where a client asks a service to perform some operation). The simplest way to do so is to use callbacks. I have made up a very simple solution to demonstrate this concept. A sample solution has two interfaces: one for the usual service contract and another one for the client callback.

[ServiceContract(CallbackContract = typeof(IContractCallback))]
public interface IContract
{
    [OperationContract]
    void Foo();
}

[ServiceContract]
public interface IContractCallback
{
    [OperationContract]
    void OnFooCallback();
}

A very basic implementation of these contract can look like this

internal class WcfService : IContract
{
    public void Foo()
    {
        //Do work...            
        var callback = OperationContext.Current
                        .GetCallbackChannel<IContractCallback>();
        callback.OnFooCallback();
    }
}

internal class ContractCallback : IContractCallback
{
    public void OnFooCallback()
    {
        Console.WriteLine("...");
    }
}

This is how this small app looks

Using the callback contract can be straight-forward, however there are some limitations:

Only NetTcpBinding, NetNamedPipeBinding and WSDualHttpBinding bindings are supported
One callback contract per service contract is allowed
Client must keep the connection open the whole time
Service must use reentrant or multiple concurrency mode

This project can be downloaded using the link below or from the github.

CallbackService.zip (17.49 kb) [Downloads: 121]

If you enjoyed this post, make sure you subscribe to my RSS feed!

WCF Client to Close or Abort

Thu, 01 Sep 2011 11:36:00 +0000

Any service oriented architecture, including WCF, follows a convention where a client implements a proxy pattern. Proxy allows the client to abstract from the concrete implementations and locations of the service. Client often creates stubs that work well in the compile time, whereas in runtime stubs are resolved and actual calls are made to the service.

Closing WCF client

A client has an inherited responsibility of gracefully closing the connection. It is always recommended to close the proxy client. If the binding between a client and a service is transport-layer sessionful, then closing a proxy is essential to tear down the connection between both parties. Service has a payload threshold defined for concurrent connections. If the number of concurrent connections goes above this threshold linearly then the overall service performance exponentially decreases. This is why it is crucial to dispose of the connection as soon as possible. Closing the proxy also notifies the service instance that it is no longer in use and may be collected by GC. If the client does not close a connection, it is still automatically torn down by WCF timeouts (found in the configuration files).

Aborting WCF client

In the situation where there is a fault in the service-client interaction, the objects on both ends are potentially totally broken. Thus using a proxy after the exception is not advised. Given the WCF binding use transport sessions, the client after a fault would not even be able to close it (if there was no transport layer session then the client could use or close the proxy, but this is not recommended as the configuration of sessions could change). So after a fault has happened the only safe operation is to abort a proxy.

Coding

Close (and Abort on fault) needs to be called for all the client calls, therefore these methods can be moved to a proxy itself

class Client1 : ClientBase<IContract1>, IContract1
{
    public void DoWork()
    {
        try
        {
            Channel.DoWork();
            Close();
        }
        catch
        {
            Abort();
            throw;
        }			
    }
}

If you enjoyed this post, make sure you subscribe to my RSS feed!

How to host many WCF services in one Windows Service

Sat, 25 Jun 2011 20:31:00 +0000

It is well known that WCF service can be hosted as a self-hosted application, IIS web service or a Windows service. This post is dedicated to the Windows service hosting and it goes through some advanced techniques on how to host many WCF services in one windows service.

So, here I have got three projects: multiple WCF services project, a client and a Windows service installer. Let us start with the WCF services. This project has three service interfaces and correspondently three service implementations.

Sample interface looks like this:

[ServiceContract]
public interface IService1
{
    [OperationContract]
    string DoAction1();
}

And its implementation is very simple

public class Service1 : WindowsManagedService, IService1
{
    public Service1() : base(typeof(Service1)) { }

    public string DoAction1()
    {
        return "Action 1";
    }
}

These are the interfaces in WCF service project

Class diagram for the services that implement those interfaces

You must have noticed that there are a few other types like IWindowsManagedService and WindowsManagedService - this is basically a bridge that allows WCF services to be hosted within Windows service.

public interface IWindowsManagedService
{
    void Start();
    void Stop();
}

public abstract class WindowsManagedService : IWindowsManagedService
{
    private ServiceHost serviceHost;

    protected WindowsManagedService(Type serviceType)
    {
        serviceHost = new ServiceHost(serviceType);
    }

    public void Start()
    {
        serviceHost.Open();
    }

    public void Stop()
    {
        serviceHost.Close();
    }
}

Next, let’s take a quick look into the host project. This project contains a configuration file for the WCF services. It also has one installer class which is picked up by installutil. But the main type here is WindowsManagedServices which uses reflection to find WCF services that can be hosted in one Windows service

public class WindowsManagedServices : ServiceBase
{
    private List<IWindowsManagedService> services;

    public WindowsManagedServices()
    {
        services = new List<IWindowsManagedService>();

        var assembly = Assembly.GetAssembly(typeof(IWindowsManagedService));
        var types = assembly.GetTypes();
        foreach (var type in types)
        {
            if (!type.IsAbstract && typeof(IWindowsManagedService).IsAssignableFrom(type))
            {
                var serviceObject = (IWindowsManagedService)Activator.CreateInstance(type);
                services.Add(serviceObject);
            }
        }
    }

    protected override void OnStart(string[] args)
    {
        foreach (var service in services)
        {
            service.Start();
        }
    }

    protected override void OnStop()
    {
        foreach (var service in services)
        {
            service.Stop();
        }
    }
}

Then, I used installutil to install the windows service and here is what I end up with

1 Windows Service WCFServices
3 WCF services
http://localhost:8083/WCFServices/IService1
http://localhost:8083/WCFServices/IService2
http://localhost:8083/WCFServices/IService3

Lastly, a sample client to this application

class Program
{
    static void Main(string[] args)
    {
        IService1 client1 = new Service1Client();
        IService2 client2 = new Service2Client();
        IService3 client3 = new Service3Client();

        Console.WriteLine("Client 1 says: " + client1.DoAction1());
        Console.WriteLine("Client 2 says: " + client2.DoAction2());
        Console.WriteLine("Client 3 says: " + client3.DoAction3());

        Console.ReadLine();
    }
}

The whole project can be downloaded using the link below. Unzip it, run the solution file and hit F5 in Visual Studio 2010. It should start two projects: services and the client.

WCFServiceHosting.zip (46.48 kb) [Downloads: 30] [UPDATED after discovery of the client's port mismatch)

If you enjoyed this post, make sure you subscribe to my RSS feed!

Why Thread.Sleep(1) takes more time?

Tue, 07 Jun 2011 14:28:00 +0000

I want to make a thread sleep for 1 ms on a Windows OS, but when I use Thread.Sleep(1) it usually takes more time, for example 10 – 20ms. This happens because desktop and server Windows are not real-time operating systems (Windows Compact Editions is). My laptop has one CPU and two cores, thus I have a very small number of independent execution units.

From the other hand there is a number of processes and many more threads. Each thread requires some processor time, that is assigned internally by the Windows scheduler. So Windows blocks all the threads but gives a certain amount of CPU time to the few selected threads, then it switches the context to the other threads. OS tries to switch the context as less as possible and stay in one context as long as possible, because switching itself doesn't do any practical payload, but still consumes the resources.

When I call Thread.Sleep, no matter how small it is, I kill the whole time span that Windows has assigned to the thread. It is clear that there are no reasons to wait for a sleeping thread. This is why the context can be safely switched straight away. It can take a few ms when the Windows returns the context control to the current thread next time and this is why I usually cannot sleep for small time span.

If I still need to sleep a very small amount of time, I can do processor spinning, e.g. System.Collections.Concurrent namespace uses spinning for non-blocking collections' updates.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Basic security for ASP.NET applications

Tue, 07 Jun 2011 11:07:00 +0000

Part 3

See Part 1 (debug, error pages, back-ups, config encryption)

See Part 2 (passwords, sensitive data, updates and patches, basic authentication)

This part concentrates on the user’s privacy, authentication, authorization and validation.

Privacy is extremely important for web application, where all the data transfer is public. It is a tall order to make private data actually private. Developers shall always assume that the web is:
- Unreliable - packets get lost, delayed, received in messed-up order etc.
- Insecure - packets travel via different countries and potentially are recorded
- Cacheable - any GET requests are possibly cached by servers and routers, thus only public data can be requested via unprotected channel
- Subject to various attacks (spoofing, denial-of-service, cross-site, man-in-the-middle and many others)
Therefore private data must be protected via encryption using only standard time-proved algorithms of layered security such as TLS (SSL), SSH, IPsec, Kerberos and others. This includes communication between client, web server, data storage, and any combination of those. Encryption certificate must be issued by a valid authority and have a valid expiry date. Sertificates shall not use compromised hash algorithms (MD5, SHA-1). Access to private keys (if used) must be strictly controlled and audited (fail).
Authentication serves the purpose to verify that you are who you claim you are. Communication between client and server must be encrypted to prevent hijacking of user name/password/security token. Encryption must include the loging page, the authentication POST destination, and the landing page. Audit of all login attempts shall be persisted to a safe protected place. Athentication can be delegated to one of the OpenID providers.
Authorization is a process of checking if the user, process, context have permissions to perform the requested action. There are various approaches to implement authorization in a web application. Authorization always occurs on the server side and shall operate in a whitelist mode - deny all operations unless allow is explicitly specified. Database users shall have a very limited amount of privileges and must not have privileges to drop a table or database. Never use a system account, such as sa for MS SQL server.
Validation for the whole web application shall be central, any income document shall get validated on the server side, e.g. XML schema verified before any processing. Server side validation is mandatory, whereas client side one is a nice optional feature to have. Failed validation shall be logged and monitored by the system administrator. HTTP verbs that modify the state (POST, PUT, PATCH, and DELETE) must be treated with additional caution. Here is a validation guidelines.

This post could not have been published without the help of Niall Merrigan

If you enjoyed this post, make sure you subscribe to my RSS feed!

Basic security for ASP.NET applications

Tue, 17 May 2011 16:03:00 +0000

Part 2

See Part 1 (debug, error pages, back-ups, config encryption)

I am continuing the series of posts about ASP.NET security and will do a little bit of basic hacking at the end today.

Most of the web applications use passwords. These passwords must not be store as is. Rather consider generating hash code or using one-way encryption. If the database is stolen (like recent incident with Sony), users passwords cannot be decrypted and thus cannot be used to get access to other systems. It’s is a well-known fact, people do not have many different passwords to access different resources
Credit cards' information is extremely sensitive data, generally it is better to avoid storing such data at all. Otherwise it is a whole new topic and totally paranoid approach to the security (which frankly goes beyond my limited knowledge)
Production servers must have recent updates and patches, any new security hole is quickly shared and attack scripts are freely distributed. Such scripts are used to perform repeated anonymous attacks
Do not use "stupid" encryption (see below)

Modern ASP.NET usually works over plain HTTP where it is possible to use Basic authentication. Basic authentication sends username and password in a virtually unencrypted way – Base64 encoded. To demonstrate its vulnerability I set up a simple ASP.NET MVC application and turned off default Forms authentication. Then, I quickly googled a Basic authentication sample. First link and I have a controller method that use Basic authentication. This is how GUI looks to the user

Seems quite secure, so I type in my user name and super-strong long password

Then I hit log in and catch the traffic using Fiddler. Let's see the request headers.

GET http://localhost:6105/secure HTTP/1.1
Host: localhost:6105
Authorization: Basic QWxpY2U6Skgsd21udmwqNjczMDVAOzt3alVXRVVO

All is good, but the user name and the password is encoded in the Authorization header. One don't need to be a magician to get the user name and the password. This functionality has been in Fiddler for a long time.

And with a few lines of code I do the same in C#

So make sure you don't use Basic authentication over plain HTTP, and better disable it in IIS.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Basic security for ASP.NET applications

Tue, 10 May 2011 10:15:00 +0000

Part 1

Security of itself is a huge topic. Every developer must know how to deal with the most widespread attacks. There are a few tips of how to write and deploy web applications that will make the attack on it less likely to succeed.

Never leave debug enabled on a production web site. It greatly affects performance and allows people to see debug information that helps to find a penetration points

<configuration>
    <system.web>
        <compilation debug="false"/>                    
    </system.web>
</configuration>

Enable error pages. Include 404 (resource not found), 500 (internal error) and general error pages. Never display actual error to the user. This information directly exposes the weaknesses of the code. Do exception logging first and use static HTML pages to display errors

<configuration>
  <system.web>
    <customErrors defaultRedirect="Error.html" mode="RemoteOnly">
      <error statusCode="500" redirect="InternalError.html"/>
      <error statusCode="404" redirect="NotFound.html"/>
    </customErrors>
  </system.web>
</configuration>

Do not leave back-ups on the web server. They can be easily downloaded.

As mentioned in [3], things can be downloaded (in contrast to served). Therefore be ready that web.config can be downloaded as it is on the site. Always encrypt sensitive information, like connection strings and application properties. Do not store user names and passwords in the web.config. This is how my connection strings section looked like before encryption (real values instead of ***)

<connectionStrings>
	<clear />	
	<add name="connName" connectionString="server=***;database=***;
        uid=***;pwd=***" providerName="***" />		
</connectionStrings>

And here is its actual look right now (user name and pwd are there, go ahead and try extracting that)

<connectionStrings configProtectionProvider="RsaProtectedConfigurationProvider">
<EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
  xmlns="http://www.w3.org/2001/04/xmlenc#">
  <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#tripledes-cbc" />
  <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
	<EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
	  <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
	  <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
		<KeyName>Rsa Key</KeyName>
	  </KeyInfo>
	  <CipherData>
		<CipherValue>c+7C9X2b2mCtOTk/kUc4+oWnJnQDQiGmTUUV7l/w0YZox/g1mFXadzmR
		WrO6M0rl5v18P5rZlG5Im0JpGKv4jB2TSXc7hedH89TFAWDkNm0t2s1kzY2JcPgJ1d1Gd
		bEYGKgsR04NyONKO/mboIF2YdzkPNdEQuQq3+I=</CipherValue>
	  </CipherData>
	</EncryptedKey>
  </KeyInfo>
  <CipherData>
	<CipherValue>MOOSknVjny5JIa+yEoSNCS44OVWovoU5RQb7hD0MXCQfBAL2AJp8R8JCuGAEFZSO
	Ms4/4Law0Pmjjkl+TsCBfy4P5UdZMVITtmpAQK5PcQx+sArCm5EcZ1TuIsrbhh8Y1jZKJjyYYsoZ1
	lx+Y2Hhw8nNsuX9t/k8bPXmdfkFAKjP7o9qkC2inASsOGyiKfOFpKpqD9A7c6Kxf/9vV0y5LbuqKF
	VcIrEGQrqXsih9T81yKLnfNxyNTyz5i3mUyvuIgw6LUkUkGY4/NhEhb8bpdPsflzSLdNlmnnwBJZf
	SAiYsL9XpjA==</CipherValue>
  </CipherData>
</EncryptedData>
</connectionStrings>

If you enjoyed this post, make sure you subscribe to my RSS feed!