Here’s a Fiddler trace of a page that uses two WCF services.

You can see there are two /js hits and they are sequential. Every visit to the same page, even with the same browser session results in making those two hits to /js. Second time when the same page is browsed:

You can see everything else is cached, except the WCF javascript proxies. They are never cached because the WCF javascript proxy generator does not produce the necessary caching headers to cache the files on browser.

Here’s an HttpModule for IIS and IIS Express which will intercept calls to WCF service proxy. It first checks if the service is changed since the cached version on the browser. If it has not changed then it will return HTTP 304 and not go through the service proxy generation process. Thus it saves some CPU on server. But if the request is for the first time and there’s no cached copy on browser, it will deliver the proxy and also emit the proper cache headers to cache the response on browser.

http://www.codeproject.com/Articles/360437/Caching-WCF-javascript-proxy-on-browser

Don’t forget to vote.

                          

See the detail implementation here:

Memory Stream Multiplexer–write and read from many threads simultaneously

Don’t forget to vote.

                          

Here’s an extension of BinaryReader for doing ReadLine over a binary stream. You can read both byte chunks, as well as read text lines at the same time.

public class LineReader : BinaryReader
{
  private Encoding _encoding;
  private Decoder _decoder;

  const int bufferSize = 1024;
  private char[] _LineBuffer = new char[bufferSize];
    
  public LineReader(Stream stream, int bufferSize, Encoding encoding)
    : base(stream, encoding)
  {
    this._encoding = encoding;
    this._decoder = encoding.GetDecoder();
  }

  public string ReadLine()
  {
    int pos = 0;
    
    char[] buf = new char[2];

    StringBuilder stringBuffer = null;
    bool lineEndFound = false;

    while(base.Read(buf, 0, 2) > 0)
    {
      if (buf[1] == '\r')
      {
        // grab buf[0]
        this._LineBuffer[pos++] = buf[0];
        // get the '\n'
        char ch = base.ReadChar();
        Debug.Assert(ch == '\n');

        lineEndFound = true;
      }
      else if (buf[0] == '\r')
      {
        lineEndFound = true;
      }          
      else
      {
        this._LineBuffer[pos] = buf[0];
        this._LineBuffer[pos+1] = buf[1];
        pos += 2;

        if (pos >= bufferSize)
        {
          stringBuffer = new StringBuilder(bufferSize + 80);
          stringBuffer.Append(this._LineBuffer, 0, bufferSize);
          pos = 0;
        }
      }

      if (lineEndFound)
      {
        if (stringBuffer == null)
        {
          if (pos > 0)
            return new string(this._LineBuffer, 0, pos);
          else
            return string.Empty;
        }
        else
        {
          if (pos > 0)
            stringBuffer.Append(this._LineBuffer, 0, pos);
          return stringBuffer.ToString();
        }
      }
    }

    if (stringBuffer != null)
    {
      if (pos > 0)
        stringBuffer.Append(this._LineBuffer, 0, pos);
      return stringBuffer.ToString();
    }
    else
    {
      if (pos > 0)
        return new string(this._LineBuffer, 0, pos);
      else
        return null;
    }
  }

}

Enjoy.

                          

Here’re the slides.

                          

Moreover, you don’t get distracted by the flashy ads and save your children and young family members from looking at foul things.

If we could get 10% of the total internet users (2bn as of Jan 2011) to save 10% CPU, power and bandwidth while browsing everyday, we could save mega watts of power everyday throughout the world!

Using this solution, you can prevent ads and tracking scripts, prevent malicious and porn websites.

How bad is it?

Let’s take an example on a popular website. The red boxes are Flash Ads (read power suckers).

Once we disable all ads and tracking scripts, here’s how it looks:

Statistics:

Not just during page loading, while you are on the page, doing nothing, just reading, browser continuously consumes CPU.

Before:

After:

Before disabling the ads and tracking scripts, CPU is always around 20-25%. After disabling it is around 8-10%. The more CPU works, the more power it consumes. If you are running on battery, you can get at least 20% more time from your battery. If you have many tabs open all the time, you can save more.

Here’s how to save CPU, bandwidth and power

Go to this website and download the HOSTS file:

http://winhelp2002.mvps.org/hosts.htm

Follow the instruction to put the HOSTS file in your C:\Windows\System32\Drivers\etc folder.

Now go to Start Menu, type Notepad but do not hit enter. Right click on Notepad and select Run As Administrator.

Go to File menu and click Open.

Copy and paste this into the File Name and click OK.

c:\windows\system32\drivers\etc\HOSTS

Now go to Edit menu and select Replace. Put 127.0.0.1 in Find box and put 255.255.255.0 in Replace box. Click Replace All.

Once done, you need to type back 127.0.0.1 for the first entry localhost.

Remember, localhost cannot be 255.255.255.0.

When you have done this correctly, it will look like this.

Save the file and exit Notepad.

Then go to Start menu and type: services.msc

From the service list, double click on “DNS Client”.

First click “Stop” to stop the service.

Then from the Startup Type dropdown, select Disabled.

Click OK.

Close all your browsers and reopen them. I highly recommend restarting your PC.

You are ready to browse faster, smarter, cheaper! 

I also highly recommend everyone to use OpenDNS. You can save yourself from getting into malicious sites and being ripped off your bank balance, property, spouse and children. Just go to www.opendns.com and follow the instruction. It is the best thing happened on the internet after Wikipedia!

How does the HOSTS file trick work?

Here’s how internet works. You type www.something.com and it goes and finds out what is the IP address for this domain. First Windows checks a file called HOSTS. If it is not defined there, it will ask the DNS Server configured for your network to give it the IP for the domain so that it can connect to the webserver. If you put fake IP in HOSTS file, Windows will hand over fake IP to the browser and browser will connect to the fake IP. Thus by putting an invalid IP, we prevent browser or any application running on your PC from reaching the ads, tracker, malicious and porn websites.

Don’t forget to share this with your friends and families!

                          

You can find details about the Project here: http://code.google.com/p/dropthings/

There are two codeproject articles that show you how it was built, tested, deployed and the production challenges I had to overcome scaling this to millions of requests per day:

Build Google IG like Portal in 7 days

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

Finally, there’s a book on it, that takes you from the initial idea to design, coding, testing, all the way to purchasing right production hardware, deployment and production troubleshooting. It is a complete end-to-end guide for a developer/startup CTO to take an idea from design to VC funded successful startup used by millions. I have captured many experiences I have learnt from my startup years at Pageflakes that I co-founded and was the founding CTO.

Building a Web 2.0 Portal with ASP.NET 3.5 from O‘Reilly.

Let’s build great web apps and save the world at the same time!

                          

Can’t wait for the day when we will be able to use C# to program DNA:

protected void CancerCell_Found(object body, CellEventArgs e)
{
  this.Attack(e.TargetCell);
}

Here’s my presentation slide. Nothing NDA or DNA in this, feel free to distribute.

                          

.DBANON=w3kYczsH8Wvzs6MgryS4JYEF0N-8ZR6aLRSTU9KwVaGaydD6WwUHD7X9tN8vBgjgzKf3r3SJHusTYFjU85y
YfnunyCeuExcZs895JK9Fk1HS68ksGwm3QpxnRZvpDBAfJKEUKee2OTlND0gi43qwwtIPLeY1;
ASP.NET_SessionId=bmnbp155wilotk45gjhitoqg; DBAUTH12=2A848A8C200CB0E8E05C6EBA8059A0DBA228FC5F6EDD29401C249D2
37812344C15B3C5C57D6B776037FAA8F14017880E57BDC14A7963C58B0A0B30229
AF0123A6DF56601D814E75525E7DCA9AD4A0EF200832B39A1F35A5111092F0805B
0A8CD3D2FD5E3AB6176893D86AFBEB68F7EA42BE61E89537DEAA3279F3B576D0C
44BA00B9FA1D9DD3EE985F37B0A5A134ADC0EA9C548D

There are 517 bytes of worthless data being sent to every css, js and images from the browser to your webserver!

You might think 517 bytes is peanut. Do the math:

• Avg page has 40 requests to server. 40 x 517 bytes = 20 KB per page view.
• 1M page views = 20 GB
• That’s 20GB of data getting uploaded to your server for just 1M page views. It does not take millions of users to produce 1M page views. Around 100k+ users using your site every day can produce 1M page views every day.

Here’s how to prevent this:

• Setup a new website and map a different subdomain to it. If your main site is www.yoursite.com then map static.yoursite.com to it.
• Manually change all the <link>, <script>, <img> css url(…) and prefix each resource with http://static.yoursite.com
• If you don’t want to do it manually, use this solution I have done before.
• Add a Global.asax and in the EndRequest do this trick:

  HttpContext context = HttpContext.Current;
  if (context.Request.Url.ToString.StartsWith("http://static.yoursite.com")
  {
    List<string> cookiesToClear = new List<string>();
    foreach (string cookieName in context.Request.Cookies)
    {
      HttpCookie cookie = context.Request.Cookies[cookieName];
      cookiesToClear.Add(cookie.Name);
    }
  
    foreach (string name in cookiesToClear)
    {
      HttpCookie cookie = new HttpCookie(name, string.Empty);
      cookie.Expires = DateTime.Today.AddYears(-1);
  
      context.Response.Cookies.Set(cookie);
    }
  }

  This code reads all the cookies it receives from request and expires them so that browser does not send those cookies again. If by any chance ASP.NET cookies get injected into the static.yoursite.com domain, this code will take care of removing them.

                          

So, you do an IIS restart. Your websites comes back online for couple of minutes and then it goes down again. After doing restart several times you realize it’s not an infrastructure issue. You have some scalability issue in your code. All the good things you have read about scalability and thought that those were fairy tales and they will never happen to you is now happening right in front of you. You realize you should have made your services async.

However, just converting your sync services to async mode does not solve the scalability problem. WCF has a bug due to which it cannot serve requests as fast as you would like it to. The thread pool it uses to handle the async calls cannot start threads as requests come in. It only adds a new thread to the pool every 500ms. As a result, you get slow rampup of threads:

Read my article to learn details on how WCF works for async services and how to fix this bug to make your async services truly async and scale under heavy load.

http://www.codeproject.com/KB/webservices/fixwcf_for_restapi.aspx

Don’t forget to vote.

                          

I have tried Scott Gu’s examples on creating RESTful routes, this MSDN Magazine article, Phil Haack’s REST SDK for ASP.NET MVC, and various other examples. But they have all made the same classic mistake – the action is part of the URL. You have to have URLs like http://localhost:8082/MovieApp/Home/Edit/5?format=Xml to edit a certain entity and define the format eg xml, that you need to support. They aren’t truly RESTful since the URL does not uniquely represent the state of an entity. The action is part of the URL. When you put the action on the URL, then it is straightforward to do it using ASP.NET MVC. Only when you take the action out of the URL and you have to support CRUD over the same URL, using three different formats – html, xml and json, it becomes tricky and you need some custom filters to do the job. It’s not very tricky though, you just need to keep in mind your controller actions are serving multiple formats and design your website in a certain way that makes it API friendly. You make the website URLs look like API URL.

The example code has a library of ActionFilterAttribute and ValurProvider that make it possible to serve and accept html, json and xml over the same URL. A regular browser gets html output, an AJAX call expecting json gets json response and an XmlHttp call gets xml response.

You might ask why not use WCF REST SDK? The idea is to reuse the same logic to retrieve models and emit html, json, xml all from the same code so that we do not have to duplicate logic in the website and then in the API. If we use WCF REST SDK, you have to create a WCF API layer that replicates the model handling logic in the controllers.

The example shown here offers the following RESTful URLs:

• /Customers – returns a list of customers. A POST to this URL adds a new customer.
• /Customers/C0001 – returns details of the customer having id C001. Update and Delete supported on the same URL.
• /Customers/C0001/Orders – returns the orders of the specified customer. Post to this adds new order to the customer.
• /Customers/C0001/Orders/O0001 – returns a specific order and allows update and delete on the same URL.

All these URLs support GET, POST, PUT, DELETE. Users can browse to these URLs and get html page rendered. Client apps can make AJAX calls to these URLs to perform CRUD on these. Thus making a truly RESTful API and website.

They also support verbs over POST in case you don’t have PUT, DELETE allowed on your webserver or through firewalls. They are usually disabled by default in most webservers and firewalls due to security common practices. In that case you can use POST and pass the verb as query string. For ex, /Customers/C0001?verb=Delete to delete the customer. This does not break the RESTfulness since the URL /Customers/C0001 is still uniquely identifying the entity. You are passing additional context on the URL. Query strings are also used to do filtering, sorting operations on REST URLs. For ex, /Customers?filter=John&sort=Location&limit=100 tells the server to return a filtered, sorted, and paged collection of customers.

Read my CodeProject article for details:

http://www.codeproject.com/KB/aspnet/aspnet_mvc_restapi.aspx

The source code is available here:

http://code.msdn.microsoft.com/Build-truly-RESTful-API-194a6253

Enjoy!

                          

<script src="someJs.js?v=1001" ></script>
<link href="someCss.css?v=2001"></link>

In order to do this, developers have to go to all the html, aspx, ascx, master pages, find all references to static files that are changed, and then increase the version number. If you forget to do this on some page, that page may break because browser uses old cached script. So, it requires a lot of regression test effort to find out whether changing some css or js breaks something anywhere in the entire website.

Another approach is to run some build script that scans all files and updates the reference to the javascript and css files in each and every page in the website. But this approach does not work on dynamic pages where the javascript and css references are added at run-time, say using ScriptManager.

If you have no way to know what javascript and css will get added to the page at run-time, the only option is to analyze the page output at runtime and then change the javascript, css references on the fly.

Here’s an HttpFilter that can do that for you. This filter intercepts any ASPX hit and then it automatically appends the last modification date time of javascript and css files inside the emitted html. It does so without storing the whole generated html in memory nor doing any string operation because that will cause high memory and CPU consumption on webserver under high load. The code works with character buffers and response streams directly so that it’s as fast as possible. I have done enough load test to ensure even if you hit an aspx page million times per hour, it won’t add more than 50ms delay over each page response time.

First, you add set the filter called StaticContentFilter in the Global.asax file’s Application_BeginRequest event handler:

Response.Filter = new Dropthings.Web.Util.StaticContentFilter(
    Response,
    relativePath =>
      {
        if (Context.Cache[physicalPath] == null)
        {
          var physicalPath = Server.MapPath(relativePath);
          var version = "?v=" +
            new System.IO.FileInfo(physicalPath).LastWriteTime
            .ToString("yyyyMMddhhmmss");
          Context.Cache.Add(physicalPath, version, null,
            DateTime.Now.AddMinutes(1), TimeSpan.Zero,
            CacheItemPriority.Normal, null);
          Context.Cache[physicalPath] = version;
          return version;
        }
        else
        {
          return Context.Cache[physicalPath] as string;
        }
      },
    "http://images.mydomain.com/",
    "http://scripts.mydomain.com/",
    "http://styles.mydomain.com/",
    baseUrl,
    applicationPath,
    folderPath);
}

The only tricky part here is the delegate that is fired whenever the filter detects a script or css link and it asks you to return the version for the file. Whatever you return gets appended right after the original URL of the script or css. So, here the delegate is producing the version as “?v=yyyyMMddhhmmss” using the file’s last modified date time. It’s also caching the version for the file to make sure it does not make a File I/O request on each and every page view in order to get the file’s last modified date time.

For example, the following scripts and css in the html snippet:

<script type="text/javascript" src="scripts/jquery-1.4.1.min.js" ></script>
<script type="text/javascript" src="scripts/TestScript.js" ></script>
<link href="Styles/Stylesheet.css" rel="stylesheet" type="text/css" />

It will get emitted as:

<script type="text/javascript" src="scripts/jquery-1.4.1.min.js?v=20100319021342" ></script>
<script type="text/javascript" src="scripts/TestScript.js?v=20110522074353" ></script>
<link href="Styles/Stylesheet.css?v=20110522074829" rel="stylesheet" type="text/css" />

As you see there’s a query string generated with each of the file’s last modified date time. Good thing is you don’t have to worry about generating a sequential version number after changing a file. it will take the last modified date, which will change only when a file is changed.

The HttpFilter I will show you here can not only append version suffix, it can also prepend anything you want to add on image, css and link URLs. You can use this feature to load images from a different domain, or load scripts from a different domain and benefit from the parallel loading feature of browsers and increase the page load performance. For example, the following tags can have any URL prepended to them:

<script src="some.js" ></script>
<link href="some.css" />
<img src="some.png" />

They can be emitted as:

<script src="http://javascripts.mydomain.com/some.js" ></script>
<link href="http://styles.mydomain.com/some.css" />
<img src="http://images.mydomain.com/some.png" />

Loading javascripts, css and images from different domain can significantly improve your page load time since browsers can load only two files from a domain at a time. If you load javascripts, css and images from different subdomains and the page itself on www subdomain, you can load 8 files in parallel instead of only 2 files in parallel.

Read here to learn how this works:

http://www.codeproject.com/KB/aspnet/autojscssversion.aspx

Appreciate your feedback.

                          

There are two steps – first configure IIS, then configure System.Net. There’s no need to tweak anything in WCF like using some Message Interceptor to inject HTTP Headers as you find people trying to do here, here and here.

Configure IIS to support gzip on SOAP respones

After you have enabled Dynamic Compression on IIS 7 following the guide, you need to add the following block in the <dynamicTypes> section of applicationHost.config file inside C:\Windows\System32\inetsrv\config folder. Be very careful about the space in mimeType. They need to be exactly the same as you find in response header of SOAP response generated by WCF services.

<add mimeType="application/soap+xml" enabled="true" />
<add mimeType="application/soap+xml; charset=utf-8" enabled="true" />
<add mimeType="application/soap+xml; charset=ISO-8895-1" enabled="true" />

After adding the block, the config file will look like this:

For IIS 6, first you need to first enable dynamic compression and then allow the .svc extension so that IIS compresses responses from WCF services.

Next you need to make WCF send the Accept-Encoding: gzip, deflate header as part of request and then support decompressing a compressed response.

Send proper request header in WCF requests

You need to override the System.Net default WebRequest creator to create HttpWebRequest with compression turned on. First you create a class like this:

public class CompressibleHttpRequestCreator : IWebRequestCreate
{
    public CompressibleHttpRequestCreator()
    {
    }

    WebRequest IWebRequestCreate.Create(Uri uri)
    {
        HttpWebRequest httpWebRequest =
            Activator.CreateInstance(typeof(HttpWebRequest),
            BindingFlags.CreateInstance | BindingFlags.Public |
            BindingFlags.NonPublic | BindingFlags.Instance,
            null, new object[] { uri, null }, null) as HttpWebRequest;

        if (httpWebRequest == null)
        {
            return null;
        }

        httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip |
            DecompressionMethods.Deflate;

        return httpWebRequest;
    }
}

Then on the WCF Client application’s app.config or web.config, you need to put this block inside system.net which tells system.net to use your factory instead of the default one.

<system.net>
  <webRequestModules>
    <remove prefix="http:"/>
    <add prefix="http:" type="WcfHttpCompressionEnabler.CompressibleHttpRequestCreator, WcfHttpCompressionEnabler, 
        Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />
  </webRequestModules>
</system.net>

That’s it.

I have uploaded a sample project which shows how all these works.

                          

First of all you should have multiple web servers behind a load balancer. This way you can take one server out of the production traffic, do your deployment and house keeping tasks like restarting IIS, and then put it back. Then you can do it for the second server and so on. This ensures there’s no outage for customer. If you can do it reasonable fast, hopefully customers won’t notice discrepancy between the servers some having new code and some having old code. You should only do this when your changes aren’t drastic. For ex, you aren’t delivering a complete revamped UI. In that case, some users hitting server1 with latest UI will suddenly get a completely different experience and then on next page refresh, they might hit server2 with old code and get a totally different experience. This works for incremental non-dramatic changes only.

During deployment you should follow these steps:

• Take server X out of load balancer so that it does not get any traffic.
• Stop all your .NET windows services on the server.
• Stop IIS.
• Delete the Temporary ASP.NET folders of all .NET versions incase you have multiple .NET versions running. You can follow this link.
• Deploy the changes.
• Flush any distributed cache you have, for ex, Velocity or Memcached.
• Start IIS.
• Start your .NET windows services on the server.
• Warm up all websites by hitting major URLs on the websites. You should have some automated script to do this. You can use tinyget to hit some major URLs, especially pages that take a lot of time to compile. Read my post on keeping websites warm with zero coding.
• Put server X back to load balancer so that it starts receiving traffic.

That’s it. It should give you a clean deployment and prevent unexpected errors. You should print these steps and hang on the desk of your deployment guys so that they never forget during deployment pressure.

Doing all these steps manually is risky. Under deployment time pressure, your production guys can make mistakes and screw up a server for good. So, I always prefer having a batch file that takes a server out and makes it ready for deploying code and then after the deployment is done, use another batch file to put the server back into load balancer traffic rotation after the server is warmed up.

Generally load balancers are configured to hit some page on your website and keep the server alive if that page returns a HTTP 200. If not, it assumes the server is dead and takes it our of rotation. For ex, say you have an alive.txt file on your website which is what load balancer is keeping an eye on. If it’s gone, the server is put out of the rotation. In that case, you can create some batch files that will take the server out, wait for couple of seconds to ensure the in-flight requests complete and then stop IIS, delete temporary ASP.NET files and make server ready to deploy stuff. Something like this:

serverout.bat
=====================
Ren alive.txt dead.txt
typeperf "\ASP.NET Applications(__Total__)\Requests Executing" -sc 30
iisreset /stop
rmdir /q /s "C:\WINDOWS\Microsoft.NET\Framework64\v1.1.4322\Temporary ASP.NET Files"
rmdir /q /s "C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files"
md "C:\WINDOWS\Microsoft.NET\Framework64\v1.1.4322\Temporary ASP.NET Files"
md "C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files"
xcacls "C:\WINDOWS\Microsoft.NET\Framework64\v1.1.4322\Temporary ASP.NET Files" /E /G MYMACHINE\IIS_WPG:F /Q
xcacls "C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files" /E /G MYMACHINE\IIS_WPG:F /Q

Similarly you should have a batch file that starts IIS, warms up some pages, and then puts the server back into load balancer.

serverin.bat
============
SET TINYGET=C:\Program Files (x86)\IIS Resources\TinyGet\tinyget.exe
iisreset /start"%TINYGET%" -srv:localhost -uri:http://localhost/ -status:200
ren dead.txt alive.txt
typeperf "\ASP.NET Applications(__Total__)\Requests Executing" -sc 30

Always try to automate this kind of admin chores. It’s difficult to do it right all the time manually under deployment pressure.

                          

Read the CodeProject article for more details.

http://www.codeproject.com/KB/webservices/quickwins.aspx

Please vote for me if you find the article useful.

                          

First you add a typical service reference and generate a ServiceReference.ClientConfig that looks like this:

<configuration>
    <system.serviceModel>
        <bindings>
            <basicHttpBinding>
                <binding name="BasicHttpBinding_ProxyService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" />
                </binding>
                <binding name="BasicHttpBinding_WidgetService" maxBufferSize="2147483647"
                    maxReceivedMessageSize="2147483647">
                    <security mode="None" /> 
                </binding>
            </basicHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:8000/Dropthings/API/Proxy.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_ProxyService"
                contract="DropthingsProxy.ProxyService" name="BasicHttpBinding_ProxyService" />
            <endpoint address="http://localhost:8000/Dropthings/API/Widget.svc/pox"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_WidgetService"
                contract="DropthingsWidgetService.WidgetService" name="BasicHttpBinding_WidgetService" />
        </client>
    </system.serviceModel>
</configuration>

As you see, all the URL are pointing to localhost, on my development environment. The Silverlight application now need to dynamically decide what URL the Silverlight app is running from and then resolve the endpoint URL dynamically.

I do this by creating a helper class that checks the URL of the Silverlight application and then decides what’s going to be the URL of the endpoint.

public class DynamicEndpointHelper
{
    // Put the development server site URL including the trailing slash
    // This should be same as what's set in the Dropthings web project's 
    // properties as the URL of the site in development server
    private const string BaseUrl = "http://localhost:8000/Dropthings/";

    public static string ResolveEndpointUrl(string endpointUrl, string xapPath)
    {
        string baseUrl = xapPath.Substring(0, xapPath.IndexOf("ClientBin"));
        string relativeEndpointUrl = endpointUrl.Substring(BaseUrl.Length);
        string dynamicEndpointUrl = baseUrl + relativeEndpointUrl;
        return dynamicEndpointUrl;
    }
}

In the Silverlight app, I construct the Service Client this way:

private DropthingsProxy.ProxyServiceClient GetProxyService()
{
    DropthingsProxy.ProxyServiceClient service = new DropthingsProxy.ProxyServiceClient();
    service.Endpoint.Address = new EndpointAddress(
        DynamicEndpointHelper.ResolveEndpointUrl(service.Endpoint.Address.Uri.ToString(), 
        App.Current.Host.Source.ToString()));
    return service;
}

After creating the service client with default setting, it changes the endpoint URL to the currently running website’s URL. This solution works when the WCF services are exposed from the same web application. If you have the WCF services hosted on a different domain and you are making cross domain calls to the WCF service then this will not work. In that case, you will have to find out what’s the domain of the WCF service and then use that instead of localhost.

                          

Caching frequently used objects, that are expensive to fetch from the source, makes application perform faster under high load. It helps scale an application under concurrent requests. But some hard to notice mistakes can lead the application to suffer under high load, let alone making it perform better, especially when you are using distributed caching where there’s separate cache server or cache application that stores the items. Moreover, code that works fine using in-memory cache can fail when the cache is made out-of-process. Here I will show you some common distributed caching mistakes that will help you make better decision when to cache and when not to cache.

Here are the top 10 mistakes I have seen:

1. Relying on .NET’s default serializer.
2. Storing large objects in a single cache item.
3. Using cache to share objects between threads.
4. Assuming items will be in cache immediately after storing it.
5. Storing entire collection with nested objects.
6. Storing parent-child objects together and also separately.
7. Caching Configuration settings.
8. Caching Live Objects that has open handle to stream, file, registry, or network.
9. Storing same item using multiple keys.
10. Not updating or deleting items in cache after updating or deleting them on persistent storage.

Let’s see what they are and how to avoid them.

http://www.codeproject.com/KB/web-cache/cachingmistakes.aspx

Please vote if you find this useful.

                          

http://www.microsoft.com/web/gallery/Dropthings.aspx

Please vote for my app. It will help spread the open source effort. 

                          

First use CamStudio to capture the screenshot into an AVI. Before you capture, you need to set the CamStudio video recording setting to one frame per second, otherwise there will be too many frames in your animated GIF. You can set it so 2 or more frames per second if you are recording some frequent changes on the screen.

This will put one frame in animated GIF per second. Since animated GIF gets pretty large due to its lossless primitive compression, you need to put as little frames on it as possible.

Now you can record screenshot using CamStudio and save it in an AVI file.

Once you have the AVI file, you need to open the AVI using Microsoft GIF animator.

Then you need to click the “Select all” button and go to Image tab and put 100 on the Duration. This will set each frame delay to 1 second, exactly what you have set in the CamStudio Video Options. If you have set 2 frames per second in CamStudio, then you need to set 50 in Microsoft GIF Animator.

Now you can save the file as an animated GIF and use it wherever you like.

I would highly recommend you further optimize the animated GIF and eliminate duplicate frames and use some advanced compression. For this you can use the ImageMagick utility. You will find various ways to optimize animated GIF on this page. I just use the following command line and it gives me pretty good output:

c:\Program Files (x86)\ImageMagick-6.6.3-Q16>convert SourceImage.gif -layers OptimizePlus DestImage.gif

                          

As your site grows from thousands to millions of users, delivering new features and bug fixes becomes an increasingly complex process where you have to ensure the performance and stability of the site is not compromised. Let me tell you some stories from the trenches how I grew my hot startup Pageflakes (acquired by MySpace founder Brad Greenspan) from thousands to millions of visits within a year surviving through production software, hardware, process, people challenges and how you can avoid and solve them using my handy collection of articles and tools.

Watch the presentation I made in a LIDNUG session:

http://www.lidnug.org/Archives.aspx

Find the session with the title “Production Challenges of ASP.NET Website”.

Unfortunately they could not record the whole session. But whatever is there should be useful for you.

                          

Let’s first identify the challenges you have in such queue tables:

• The table is both read and write. Thus queuing and dequeuing impact each other and cause lock contention, transaction deadlocks, IO timeouts etc under heavy load.
• When multiple receivers try to read from the same queue, they randomly get duplicate items picked out of the queue, thus resulting in duplicate processing. You need to implement some kind of high performance row lock on the queue so that same item never gets picked up by concurrent receivers.
• The Queue table needs to store rows in certain order and read in certain order, which is an index design challenge. It’s not always first in and first out. Sometimes Orders have higher priority and need to be processed regardless of when they are queued.
• The Queue table needs to store serialized objects in XML or binary form, which becomes a storage and index rebuild challenge. You can’t rebuild index on the Queue table because it contains text and/or binary fields. Thus the tables keep getting slower and slower every day and eventually queries start timing out until you take a downtime and rebuild the indexes.
• During dequeue, a batch of rows are selected, updated and then returned for processing. You have a “State” column that defines the state of the items. During dequeue, you select items of certain state. Now State only has a small set of values eg PENDING, PROCESSING, PROCESSED, ARCHIVED. As a result, you cannot create index on “State” column because that does not give you enough selectivity. There can be thousands of rows having the same state. As a result, any dequeue operation results in a clustered index scan that’s both CPU and IO intensive and produces lock contention.
• During dequeue, you cannot just remove the rows from table because that causes fragmentation in the table. Moreover, you need to retry orders/jobs/notification N times  incase they fail on first attempt. This means rows are stored for longer period, indexes keep growing and dequeue gets slower day by day.
• You have to archive processed items from the Queue table to a different table or database, in order to keep the main Queue table small. That means moving large amount of rows of some particular status to another database. Such large data removal leaves the table highly defragmented causing poor queue/dequeue performance.
• You have a 24×7 business. You have no maintenance window where you can take a downtime and archive large number of rows. This means you have to continuously archive rows without affecting production queue-dequeue traffic.

If you have implemented such queue tables, you might have suffered from one or more of the above challenges. Let me give you some tips on how to overcome these challenges and how to design and maintain a high performance queue table.

Read the article for details:

http://www.codeproject.com/KB/database/fastqueue.aspx

Please vote if you find this useful.

                          

Solution is to use OUTER APPLY.

Consider some tables like this:

Customer Table

Contact table

We need to create a flat file export from this where the output needs to look like:

There are some complex requirement in the output:

• If customer has multiple phone, then it needs to select the one which is flagged as primary.
• If customer has no home address, then it needs to show “No home address” instead of NULL.
• It needs to tell if the phone address we got is primary phone or not.

The query to generate this will be:

SELECT 
    c.CustomerID,
    c.FirstName,
    c.LastName,
    c.DOB,

    'HomeAddress' = 
    CASE 
        WHEN home.ContactValue IS NULL THEN 'No Home Address'
        ELSE home.ContactValue
    END,
    work.ContactValue,
    phone.ContactValue as PrimaryPhone,
    'IsPhonePrimary' = 
    CASE 
        WHEN phone.IsPrimary = 1 THEN 'Yes'
        ELSE 'No'
    END
FROM Customer c

OUTER APPLY (
    SELECT TOP 1 ContactValue from Contact WHERE CustomerID = c.CustomerID
    AND ContactType = 'HomeAddress'
    ORDER BY IsPrimary DESC
) AS home

OUTER APPLY (
    SELECT TOP 1 ContactValue from Contact WHERE CustomerID = c.CustomerID
    AND ContactType = 'WorkAddress'
    ORDER BY IsPrimary DESC
) AS work

OUTER APPLY (
    SELECT TOP 1 ContactValue, IsPrimary from Contact WHERE CustomerID = c.CustomerID
    AND ContactType = 'Phone'
    ORDER BY IsPrimary DESC
) AS phone

All the tricks are in the OUTER APPLY blocks. OUTER APPLY selects the row that needs to appear as the value of the columns in the output after the customer table fields.

The primary address is selected by reverse ordering the rows selected from Contact table by IsPrimary field. Thus the rows having True comes first.

                          

Read how to build such a diagnostics page:

http://www.codeproject.com/KB/aspnet/selfdiagnostics.aspx

Vote for me if you find this useful.

                          

Read this article for details:

http://www.codeproject.com/KB/aspnet/watinajaxtest.aspx

I have written some extension methods for WatiN to help facilitate AJAX related tests, especially with ASP.NET UpdatePanel, jQuery and dealing with element positions. I will show you how to use these libraries to test sophisticated AJAX websites, like the one I have built – Dropthings, which is a widget powered ASP.NET AJAX portal using ASP.NET UpdatePanel, jQuery to create a Web 2.0 presentation. You can simulate and test UpdatePanel updates, AJAX calls and UI update and even drag & drop of widgets!

You can see the implementation of automated tests in my open source project codebase.

Tests written using WatiN replaces the need for human driven tests and thus sheds significant time off your regular regression test suite. Moreover, it empowers developers with a way to quickly run regression tests whenever they need to, without waiting for human QA resource’s availability. When you hook it with xUnit like test frameworks and integrate with your continuous build, you can run UI tests automatically to test all the UI scenarios overnight after your nightly build and generate reports without requiring any manual intervention.

                          

Unlike user stories, the “Behavior” suggested by Behavior Driven Development (BDD) is a much better approach because the format of a behavior (Givencontext, When event, Then outcome), when used correctly, lets you think in terms of sequence of events, where the context, event and outcome are captured for each and every action user or system does, and thus works as a definite spec for designing the UI and architecture. It follows the Cause and Effect model, thus can explain how the world (or your software) works. It can be so precise that sometimes a behavior work as guideline for a developer to write a single function! Not just the develoeprs, even the QA team can clearly capture what action they need to perform and how the system should respond. However, to get the real fruit out of behaviors, you need to to write them properly, following the right format. So, let me give you some examples on how you can write good behaviors for UI, business layer, services and even functions and thus eliminate repeated requirement analysis that usually happens throughout the user-story driven development lifecycle.

Read more about how user stories suck and if behavior is used throughout the development lifecycle, it can greatly reduce repeated requirement analysis effort and can make the communication between product, design, development and QA team much more effective:

http://www.codeproject.com/KB/architecture/userstorysucks.aspx 

If you like it, vote for it!

                          

http://www.codeproject.com/KB/linq/ef.aspx

In .NET 4.0, most of the problems are solved, but not all. So, you should read this article even if you are coding in .NET 4.0. Moreover, there’s enough insight here to help you troubleshoot EF related problems.

You might think “Why bother using EF when Linq to SQL is doing good enough for me.” Linq to SQL is not going to get any innovation from Microsoft anymore. Entity Framework is the future of persistence layer in .NET framework. All the innovations are happening in EF world only, which is frustrating. There’s a big jump on EF 4.0. So, you should plan to migrate your L2S projects to EF soon.

                          

I did some CPU profiling on my open source project Dropthings and found that the highest CPU is consumed by Unity’s Resolve<>(). There’s no funky use of Unity in the project. Straightforward Register<>() and Resolve<>(). But as you can see, Resolve<>() is consuming significantly high CPU even after the site is warm and has been running for a while.

Then I tried Munq, which is a basic Dependency Injection Container. It has everything you will usually need in a regular project. It boasts to be the fastest DI out there. So, I converted all Unity code to Munq in Dropthings and did a CPU profile and Whala!

There’s no trace of any Munq calls anywhere. That proves Munq is a lot faster than Unity.

                          

First get tinyget from here. Download and install the IIS 6.0 Resource Kit on some PC. Then copy the tinyget.exe from “C:\Program Files (x86)\IIS Resources\TinyGet\” to the server where your IIS 6.0 or IIS 7 is running.

Then create a batch file that will hit the pages and webservices. Something like this:

SET TINYGET=C:\Program Files (x86)\IIS Resources\TinyGet\tinyget.exe

"%TINYGET%" -srv:dropthings.omaralzabir.com -uri:http://dropthings.omaralzabir.com/ -status:200
"%TINYGET%" -srv:dropthings.omaralzabir.com -uri:http://dropthings.omaralzabir.com/WidgetService.asmx?WSDL - status:200

First I am hitting the homepage to keep the webpage warm. Then I am hitting the webservice URL with ?WSDL parameter, which allows ASP.NET to compile the service if not already compiled and walk through all the operations and reflect on them and thus loading all related DLLs into memory and reducing the warmup time when hit.

Tinyget gets the servers name or IP in the –srv parameter and then the actual URI in the –uri. I have specified what’s the HTTP response code to expect in –status parameter. It ensures the site is alive and is returning http 200 code.

Besides just warming up a site, you can do some load test on the site. Tinyget can run in multiple threads and run loops to hit some URL. You can literally blow up a site with commands like this:

"%TINYGET%" -threads:30 -loop:100 -srv:google.com -uri:http://www.google.com/ -status:200

Tinyget is also pretty useful to run automated tests. You can record http posts in a text file and then use it to make http posts to some page. Then you can put matching clause to check for certain string in the output to ensure the correct response is given. Thus with some simple command line commands, you can warm up, do some transactions, validate the site is giving off correct response as well as run a load test to ensure the server performing well. Very cheap way to get a lot done.

                          

using(var client = new SomeClient()) {

. 

.

.

}

Or, if you are doing it the hard and slow way (without really knowing why), then:

using(var factory = new ChannelFactory<ISomeService>()) {

var channel= factory.CreateChannel();

.

.

.

}

That’s what we have all learnt in school right? We have learnt it wrong!

When there’s a network related error or the connection is broken, or the call is timed out before Dispose is called by the using keyword, then it results in the following exception when the using keyword tries to dispose the channel:

failed: System.ServiceModel.CommunicationObjectFaultedException : 
The communication object, System.ServiceModel.Channels.ServiceChannel, 
cannot be used for communication because it is in the Faulted state.
    
    Server stack trace: 
    at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
    
    Exception rethrown at [0]: 
    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at System.ServiceModel.ICommunicationObject.Close(TimeSpan timeout)
    at System.ServiceModel.ClientBase`1.System.ServiceModel.ICommunicationObject.Close(TimeSpan timeout)
    at System.ServiceModel.ClientBase`1.Close()
    at System.ServiceModel.ClientBase`1.System.IDisposable.Dispose()

There are various reasons for which the underlying connection can be at broken state before the using block is completed and the .Dispose() is called. Common problems like network connection dropping, IIS doing an app pool recycle at that moment, some proxy sitting between you and the service dropping the connection for various reasons and so on. The point is, it might seem like a corner case, but it’s a likely corner case. If you are building a highly available client, you need to treat this properly before you go-live.

So, do NOT use using on WCF Channel/Client/ChannelFactory. Instead you need to use an alternative. Here’s what you can do:

First create an extension method.

public static class WcfExtensions
{
    public static void Using<T>(this T client, Action<T> work)
        where T : ICommunicationObject
    {
        try
        {
            work(client);
            client.Close();
        }
        catch (CommunicationException e)
        {
            client.Abort();
        }
        catch (TimeoutException e)
        {
            client.Abort();
        }
        catch (Exception e)
        {
            client.Abort();
            throw;
        }
    }
}

Then use this instead of the using keyword:

new SomeClient().Using(channel => {
    channel.Login(username, password);
});

Or if you are using ChannelFactory then:

new ChannelFactory<ISomeService>().Using(channel => {    
    channel.Login(username, password);
});

                          

Watch the screencast here:

The next video tutorial is about doing BDD to do automated UI tests. It shows how test engineers can take behaviors and then write tests that tests a prototype UI in isolation (just like Service Contract) in order to ensure the prototype conforms to the expected behaviors, while developers can write the real code and build the real product in parallel. When the real stuff is done, the same test can test the real stuff and ensure the agreed behaviors are satisfied. I have used WatiN to automate UI and test UI for expected behaviors.

Hope you like it!

                          

Tips and Tricks to rescue overdue projects

Next one is about convincing management to go for automated test and give developers extra time per sprint, at the cost of reduced productivity for couple of sprints. It’s hard to negotiate this with even dev leads, let alone managers. Whenever you tell them – there’s going to be less features/bug fixes delivered for next 3 or 4 sprints because we want to automate the tests and reduce manual QA effort; everyone gets furious and kicks you out of the meeting. Especially in a startup where every sprint is jam packed with new features and priority bug fixes to satisfy various stakeholders, including the VCs, it’s very hard to communicate the benefits of automated tests across the board. Let me tell you of a story of one of my startups where I had the pleasure to argue on this and came out victorious.

How to convince developers and management to use automated test instead of manual test

If you like these, please vote for me!