Just Another IIS Blog

IIS7 and Failover Clustering

thomad — Tue, 27 Oct 2009 07:00:00 GMT

We recently published an article on how to enable failover clustering for IIS7 and Microsoft Cluster Service (MSCS). MSCS provides failover and increased availability of applications by failing over to a second machine that is on stand-by or by just taking the sick machine out of the network so that no requests are routed to it anymore.

There is a fundamental problem with configuring failover clustering for IIS however: IIS in itself is not an application; IIS is an application platform and it's hard to come up with a clear yay or nay when to fail over. We don't know enough to decide if an application is terminally sick and can't recover. Is it when a web application returns 500 errors, when the application is overloaded and TCP connections can't be established anymore, when its Application Pool crashes repeatedly? What if more than one application is hosted on the IIS machine? Should IIS only fail over when W3SVC is terminated? Should IIS fail over if one of the web applications is dead, when 50% of the apps are dead? 90%?

There is no good way to make these decisions without involving the people managing the IIS machine and the web application developers that run code on the IIS box. And that's why we decided to publish sample code that somebody can take and adjust for their specific needs.

The sample code in the article is a script that can be configured as a Generic Script Resource in the MSCS service. The script monitors the run-time status of the IIS7 "Default Web Site" and IIS7 "DefaultAppPool" and will trigger a fail over if one of them is stopped. It is trivial to change the Site or Application Pool name in the script and it should be easy for a developer familiar with VBScript to add some custom functionality to this script.

Happy fail over!

Kerberos Authentication Issues

thomad — Sat, 24 Oct 2009 05:30:58 GMT

We ran into some problems with Kerberos authentication lately and this forced me to unearth the knowledge I once had about Kerberos.

The Issue:
Some Microsoft employees ran into 400 errors when accessing certain Intranet web pages. A 400 error is pretty much a bad request. Kerberos authentication was the cause for this problem.

The Details:
The web sites in question returned a "400 error - request too long" for content protected with IIS Windows Integrated Authentication. It turns out that people who faced this problem had an Authorization header that was larger than 16kB and that HTTP.SYS (the kernel-mode driver underneath IIS) has a request limit of 16kB, i.e. no request can be larger than 16kB by default.

The problem and the fix - increasing the maximum header size and the maximum request size of HTTP.SYS - is described in this KB article. So why would the Kerberos Authorization get larger than 16kB? Turns out that the Kerberos ticket which is encoded in the Authorization header contains all group memberships of the user who wants to authenticate against the web server. The more groups a user is a member of the larger the Authorization header gets. The magic number of group memberships seems to be around 300.

So here are a couple of ways to investigate the problem if you think you run into similar issues:

1) The whoami tool can tell you your group memberships:

whoami /groups

2) Here is a piece of ASP.NET code that tells you how big the authorization header is and the group memberships of the authenticating user (just save the file as kerb_check.aspx in your wwwroot directory and turn Windows Integrated authentication on and anonymous authentication off):

<%@Language ="C#"%>
<%  
    int authHeaderSize = 0; 
    string username = Request.ServerVariables["AUTH_USER"]; 
    if (username == "")
    {
        Response.Write("<br/>Authentication is not enabled");    
    }
    else
    {
        authHeaderSize = Request.Headers["Authorization"].Length;
        Response.Write("<h2>Size of " + 
                        username + 
                        "'s Authorization header: <br/><b>" + 
                        authHeaderSize + 
                        "</b> bytes. <p/></h2>");
        Response.Write("<h2><br/><b>Group Memberships for user " + username + ":</b></h2>");
        int count=0;
        System.Security.Principal.WindowsIdentity winId = 
                (System.Security.Principal.WindowsIdentity)HttpContext.Current.User.Identity;
        foreach (System.Security.Principal.IdentityReference ir in winId.Groups)
        {
            try
            {
                Response.Write("<br/>" + 
                               ((System.Security.Principal.NTAccount)ir.Translate
                               (typeof(System.Security.Principal.NTAccount))).Value);   
                count++;   
            }
            catch (Exception inner)
            {
                Response.Write("<br/> --- cannot resolve group ---");
            }
        }
        Response.Write("<br/><br/><b>You are member of " + count + " groups.</b>"); 
    }   
 %>

Session-based Authentication
Another problem I discovered while looking at the issue: when Kerberos authentication is used IIS7 and IIS 7.5 force the re-authentication of every request. This is unfortunate because it doesn't scale well. Just imagine a single html page with 50 images. If your Kerberos header is 10kB Internet Explorer sends 510kB (51 requests x10kB authorization header) only for authentication purposes.

When NTLM is used on the other hand, session-based authentication is enabled. If session-based authentication is enabled authentication only has to be done per TCP/IP session. This scales much better because Internet Explorer usually creates only 2 to 6 TCP/IP connections to a web server. HTTP requests are then done within these TCP/IP sessions, i.e. the connections are reused to make multiple requests. So if session-based authentication is enabled instead of sending 510kb authentication headers only 20-60kb authentication headers will be sent by IE.

I'm still investigating why session-based authentication is not used for Kerberos - I hope there is a good reason for it :). There is a configuration property in IIS7.x that allows you to switch Kerberos to session-based authentication. It is the AuthPersistNonNTLM property. If you want to turn on server-wide session-based authentication for Kerberos here is your command:


%windir%\system32\inetsrv\appcmd.exe set config -section:windowsAuthentication /authPersistNonNTLM:"True" /commit:apphost

That's it for today.

Thomas

Now Available: The IIS 7.5 Application Warm-Up Module

thomad — Wed, 14 Oct 2009 21:15:31 GMT

IIS is a demand-driven web server, i.e. IIS does things only when asked for. For example: an IIS worker process spawns up only when requests arrive for the sites that are hosted in this worker process. Without requests there isn't a worker process. This is great from a resource consumption point of view. Worker processes which don't run do not consume resources, memory in particular.

There is a drawback to this architecture however. The first requests that get handled by a newly spawned worker process might have to wait longer due to the initialization costs of the web application(s) living within the worker process. Typical example of initialization activities are:

Initialization of data structures
Loading data from a datastore into memory (caching)
Compilation of code, e.g. .NET applications
Establishing database connections

For big applications the initialization phase might take considerable time and IT administrators might not want their customers to pay the price for the initialization.

To fix this problem IIS 7.5, which ships as part of Windows Server 2008 R2 and Windows 7, comes with a new capability that allows the warm-up of initialization-heavy web applications.

The IIS Warm-Up module allows administrators or web application developers to configure a collection of URLs which will be executed before IIS accepts user requests for the web application in question. By executing the configured warm-up URLs data structures and caches will be initialized and requests arriving from the network will not experience any initialization delays.

More information and the download locations for the Application Warm-Up for IIS 7.5 can be found on http://www.iis.net/extensions/ApplicationWarmUp

What account is your AppPool running as?

thomad — Sat, 28 Mar 2009 15:23:10 GMT

IIS 7.5 on Windows 7 and Windows Server 2008 R2 is changing the default Application Pool identity from NetworkService to virtual accounts with the name of the Application Pool itself. The Application Pool ‘DefaultAppPool’ will run as the virtual account with the name ‘DefaultAppPool’. Given this change I thought it might be useful to have a little script that lists all Application Pools and the accounts they are running as. Just copy the following script and save it as myAppPools.js.

var ahadmin = new ActiveXObject("Microsoft.ApplicationHost.AdminManager");

var APPPOOLSSECTION = "system.applicationHost/applicationPools";
var APPHOSTROOT     = "MACHINE/WEBROOT/APPHOST";

Main();

function Main()
{
    try
    {
       var appPoolsSection = ahadmin.GetAdminSection(APPPOOLSSECTION, APPHOSTROOT);
       WScript.Echo ("Available Application Pools and Identities");
       WScript.Echo ("==========================================");

       for (var i=0;i<appPoolsSection.Collection.Count;i++)
       {
          var appPool = appPoolsSection.Collection.Item(i);         
          var appPoolName = appPool.Properties.Item("name").Value;
          var processModel = appPool.GetElementByName("processModel");
          if (processModel.Properties.Item("identityType").Value == 0) //LocalSystem
          {            
             WScript.Echo ("APPPOOL:     \t" + appPoolName);
             WScript.Echo ("IDENTITYTYPE:\tLocalsystem");
             WScript.Echo ("USER:        \tLocalSystem");
          }
    
          if (processModel.Properties.Item("identityType").Value == 1) //LocalService
          {            
             WScript.Echo ("APPPOOL:     \t" + appPoolName);
             WScript.Echo ("IDENTITYTYPE:\tLocalService");
             WScript.Echo ("USER:        \tLocalService");
          }

          if (processModel.Properties.Item("identityType").Value == 2) //NetworkService
          {            
             WScript.Echo ("APPPOOL:     \t" + appPoolName);
             WScript.Echo ("IDENTITYTYPE:\tNetworkService");
             WScript.Echo ("USER:        \tNetworkService");
          }
          if (processModel.Properties.Item("identityType").Value == 3) //Specific User
          {            
             WScript.Echo ("APPPOOL:     \t" + appPoolName);
             WScript.Echo ("IDENTITYTYPE:\tSpecific User");
             WScript.Echo ("USER:        \t" + processModel.Properties.Item("userName").Value);
          }
          if (processModel.Properties.Item("identityType").Value == 4) //AppPool identity
          {            
             WScript.Echo ("APPPOOL:     \t" + appPoolName);
             WScript.Echo ("IDENTITYTYPE:\tAppPoolIdentity");
             WScript.Echo ("USER:        \t" + appPoolName);
          }
          WScript.Echo("\n");
       }
    }
    catch (e)
    {
       WScript.Echo("Script failed" + e.number);
       WScript.Echo(e.description);
    }
}

Now available for download: Release Candidate of IIS PowerShell Snap-in

thomad — Fri, 16 Jan 2009 18:28:41 GMT

We just made the Release Candidate of the IIS PowerShell Snap-in available. A lot of work was done between Tech Preview 2 and now. We focused mainly on augmenting the PowerShell Provider with almost 70 task-oriented cmdlets useful for day-to-day administrative tasks. Here is a quick categorization of the task-oriented cmdlets:

Mangement of Sites/Apps/Vdirs/AppPools (create/start/stop/remove/convert/recycle)
Management of site bindings (add/change/remove)
Management of Handlers and Modules/Managed Modules (add/change/remove)
Enablement of Request Tracing
Backup and Restore of IIS configuration
Locking/unlocking of sections/elements/attributes
Getting file system path for config files and web application content
Access to run-time data, e.g.
---Exploring currently executing requests
---AppDomain Management

I added a new walkthrough that uses some of the new task-oriented cmdlets. But there is an easier way to get started if you don’t want to read through the walkthrough. Just list the available cmdlets with

get-command -pssnapin WebAdministration

This will display all the cmdlets available in the IIS7 PowerShell Snap-in. To find help on how to use a particular cmdlet you just have to type

get-help <cmdletname>

For example:

get-help New-WebSite

get-help <cmdletname> –example

if you are only interested in an example how to use the cmdlet.

Have fun!

CTP2 of IIS7 PowerShell Provider Released!

thomad — Thu, 03 Jul 2008 17:06:00 GMT

Tech Preview 2 of the IIS7 PowerShell Provider is now available for download.

What’s new in Tech Preview 2?

IIS7 Powershell provider now supports SSL (installing and acquiring a certificate, creating an ssl site binding)
Tech Preview 2 ships with 40 new cmdlets. All of these cmdlets are for day-to-day IIS tasks like creating web-sites, web-applications, enabled request tracing, adding a handler or a module. The complete list is at http://learn.iis.net/page.aspx/492/using-the-task-based-cmdlets

IIS7 PowerShell Provider features:

Create Web-Sites, Web Applications, Virtual Directories and Application Pools
Change Simple Configuration Properties on Web-Sites, Application Pools, Web Applications and Virtual Directories
Add and Change Complex Configuration Settings
Query Run-time Data (Web-Site State, Application Pool State, Currently Executing Requests)
Execute Advanced Configuration Tasks, Scripting, Integration with other PowerShell Snap-Ins and features
Search and Discover Configuration Settings

DOWNLOAD:
Tech Preview 2 of the IIS 7.0 PowerShell Provider can be found here:

FORUMS:
Go to our PowerShell forum if you need support or if you are looking for 'Tips and Tricks'
http://forums.iis.net/1151.aspx

WALKTHROUGHS
The walkthroughs are here: http://learn.iis.net/page.aspx/447/managing-iis-with-the-iis-70-powershell-provider/

Now Available: URL Rewriter Tech Preview 1

thomad — Sat, 31 May 2008 00:31:00 GMT

Today we are releasing Technical Preview 1 of the URL Rewrite Module for IIS 7.0.

The URL Rewrite Module provides a rule-based rewriting mechanism for changing request URL’s before they get processed by IIS. The module supports regular expression based URL rewriting logic or a simpler wildcard-based URL rewriting logic. Rewriting decisions can be based on the URL, HTTP headers and server variables. While the primary purpose of the module is to rewrite URLs, it also has functionality to perform redirects, send custom responses and abort requests based on the logic expressed in the rewrite rules.

Install the URL Rewrite Module today!

Microsoft URL Rewrite Module for IIS 7.0 CTP1 (x86)

Microsoft URL Rewrite Module for IIS 7.0 CTP1 (x64)

Feature Set

Here is a quick feature overview:

Rules-based URL rewriting engine. Rules are used to compare/match the request URL with and what to do if comparison was successful.

Regular expression pattern matching. Rewrite rules can use ECMA-262 compatible regular expression syntax for pattern matching.
Wildcard pattern matching. Rewrite rules can use Wildcard syntax for pattern matching

Back-references to patterns and conditions. Back-references are used to capture parts of a matched URL so that it can be re-used later in a rule when constructing a substitution URL string. Back-references are available with regular expression and wildcards patterns.
Global and distributed rewrite rules. Global rules are used to define server-wide URL rewriting logic. Global rules cannot be overridden or disabled by lower configuration levels. Distributed rules are used to define URL rewriting logic specific to a particular configuration scope, e.g. an web application.
Access to server variables and http headers. Server variables and HTTP headers provide additional information about current HTTP request. This information can be used to make rewriting decisions or to compose the output URL.
Various rule actions. Instead of rewriting a URL, a rule may perform other actions, such as issue an HTTP redirect, abort the request, or send a custom status code to HTTP client.
Rewrite maps. Rewrite map is an arbitrary collection of name-value pairs that can be used within rewrite rules to generate the substitution URL during rewriting. Rewrite maps are particularly useful when you have a large set of rewrite rules, all of which use static strings (i.e. there is no pattern matching used). In those cases, instead of defining a large set of simple rewrite rules, you can put all the mappings between input URL and substitution URL as keys and values into the rewrite map, and then have one rewrite rule which references this rewrite map to look up substitution URL based on the input URL.
UI for managing rewrite rules. Rewrite rules can be added, removed and edited by using "URL Rewrite Module" feature in IIS Manager.
GUI tool for importing of mod_rewrite rules. URL rewrite module includes a GUI tool for converting rewrite rules from mod_rewrite format into IIS format.

More information

URL Rewrite Module Walkthroughs - http://learn.iis.net/page.aspx/460/using-url-rewrite-module/

URL Rewrite Module Configuration Reference - http://learn.iis.net/page.aspx/465/url-rewrite-module-configuration-reference/

Questions and feedback

URL Rewrite Forum - http://forums.iis.net/1152.aspx

IIS Discussion Digest: Edition 1

thomad — Sun, 11 May 2008 00:43:00 GMT

Many Microsoft folks use an internal alias to discuss their IIS issues. Each week plenty of interesting topics come up. I will try to pick the most interesting topics and blog them regularly. I was a bit late this week, but nonetheless here is Edition Numero One:

From: MF
Subject: RE: About HTTP Compression

Hi All,

I remember not long ago there was a thread about what would be the minimum size of an output in order to take any advantage of having compression enabled. Can somebody kindly forward that thread to me? Or let me know what was the conclusion on that?

Tnks,
M.

From: JJ
Subject: RE: About HTTP Compression

Well, the simplistic answer is: if it is less than 2.8k compression doesn’t help. Why 2.8k? Because default packet size is 1.4k with a minimum packets per send of 2. So if you try to optimize for less than 2.8 k, you aren’t really doing anything because you aren’t changing the number of packets – which is what drives round trips.
Beyond that statement things get a bit confusing, because you never know how many packets the server is going to send (thanks to NT’s TCP implementation and scalable windows sizes, etc).

From: TK
Subject: IIS 6 Queuing from HTTP.SYS
Delivering some IIS training at present, and had a couple of questions I'm hoping to get some feedback on around Recycling and HTTP.SYS Queuing:
- When recycling a healthy process, is the terminating process able to send any outstanding responses after the new worker process is established? Or is the point at which the new process is registered the point at which the old HTTP.SYS response queue is invalidated? I suspect it's alive for the lifetime of the process, but I'm still reading the docs.
- When recycling a hung application pool, does the new worker process inherit the requests currently in the HTTP.SYS queue for that namespace (assuming the app pool hasn't been picking requests up from the kernel queue for a while), or does the new w3wp only pull new requests from after it starts up?
Thanks!

From: AK
Subject: RE: IIS 6 Queuing from HTTP.SYS
1) WAS recycling works be creating a new Worker Process including its queue and http.sys routing new requests to that queue. The "old" WP is left for draining its queue (unless a timeout is reached… in that case WAS terminates the lingering WP).
2) The above is called “overlapped” recycling. You can (but shouldn’t IMHO) configure non-overlapped recycling. In that case the old WP termination is waited for before a new Worker Process is spun up. I don’t know by heart who owns the queuing of the requests coming in during the time window where the old WP is starting shutdown and the new didn’t yet start.
I hope this helps.
ciao,
A.

From: TD
Subject: RE: IIS 6 Queuing from HTTP.SYS
2) Regarding "who is queuing requests coming in during the time window where the old WP is starting shutdown":
It's HTTP.SYS. When IIS starts (WAS in IIS7 or W3SVC in IIS6) it reads configuration and configures HTTP.SYS. HTTP.SYS creates a request queue and starts listening on the network for requests. No worker processes are running. HTTP.SYS asks WAS/W3SVC to start a worker process when the first request arrives. Once the worker process is running it picks up requests from the HTTP.SYS request queue. Requests in the request queue don't get lost if the worker process is recycled or simply crashes. They simply sit in the HTTP.SYS request queue and wait until a new worker process picks them up.
T.

From: AT
Subject: RE: IIS 6 Queuing from HTTP.SYS
Above is true unless the request is already dispatched to a worker process and the Worker Process was in the middle of sending the response when it crashed. Otherwise associated connection will be aborted. As it is not possible to recover after the Worker Process started sending the response. This is also one of the reasons why draining is necessary as indicated by A. above.

The IIS Process Model Features

thomad — Thu, 08 May 2008 06:08:48 GMT

A process model is needed to run more than one web-site, web application or web-service securely and reliably on a single machine. In Shared Hosting scenarios hundreds or even thousands of web-sites run on an individual machine. The code running on these web-sites is usually not well tested, if at all. Without a powerful process model the result would be extremely poor reliability. But a process model not only guarantees availability; it also needs to isolate them so that individual web applications don't interfere with each other.

The Windows Process Activation Service (WAS) is the system component providing the Process Model in IIS 7.0. This article discusses the WAS features and functions.

1. Application Pools

Processes are the containers that provide isolation and security boundaries in the Windows Operating System. Consequently web applications or web-sites have to run in separate processes in order to achieve isolation between them. In the IIS terminology the management unit for these separate IIS processes is called 'Application Pool'.

An Application Pool can be configured to run a group of URLs, i.e. web-sites, web applications or web-services. When a client requests one of these Application Pool URLs an IIS worker process (w3wp.exe) is spawned by WAS to execute the code necessary to send a response. WAS's main task is to manage Application Pools - WAS spawns worker processes, monitors their health, recycles them if necessary and makes sure none of them consume more resources than specified in the corresponding AppPool configuration. WAS is also the arbiter and collector for run-time and state data, e.g. performance counters, site and Application Pool state.

2. Architectural Diagram

The diagram below shows the core pieces of the IIS7 architecture.

Configuration is stored in the applicationhost.config file (left).
HTTP.SYS is a kernel-mode component that listens to the network, accepts connections, assigns requests into Application Pool queues and queues these requests if no IIS worker process is running. HTTP.SYS also caches responses and does SSL in IIS 7.0
The worker process on the right hosts all custom code (e.g. ASP and ASP.Net pages, modules, ISAPI filter and extensions etc.). There can be hundreds of these worker processes depending on how the IIS Administrator decides to isolate his web-sites and applications. IIS worker processes pick up waiting requests from the corresponding HTTP.SYS request queue.
The Windows Activation Service is a system service that runs in SVCHOST.EXE. WAS reads configuration from applicationhost.config, reacts to configuration changes, manages worker processes, controls their live time and health, recycles them if necessary and prevents resource exhaustion. W3SVC is another service living in the same SVCHOST.EXE as WAS. W3SVC hosts the HTTP specific part of the IIS process model; W3SVC configures HTTP.SYS with the URLs to listen on and is called by HTTP.SYS if requests arrive and worker processes are needed.

3. Process Model Features

Web-Sites and applications get the following benefits by running in the WAS provided infrastructure:

Efficient Resource Management

On-Demand Activation

Resources like RAM and CPU are scarce in multi-tenant scenarios. WAS will start an IIS worker process only once requests for a particular web site or web application arrive.

Idle-timeout

Because resources are usually scarce WAS can shutdown web applications based on a configurable idle-timeout.

Health Monitoring

To ensure their health WAS monitors the worker processes it spawned. Health messages are periodically sent to each running worker process. If the worker process doesn't respond in a configurable time interval the worker process will be recycled or killed. This way undetected deadlocks in worker processes get automatically fixed by restarting the worker process.

Startup Limit

Part of the Rapid-Fail Protection feature is the Startup limit. If a worker process doesn't report back to WAS within the configurable startup-limit it will be killed and the Rapid-Fail-Protection counter is incremented. Application Pools are stopped, i.e. restarting the worker process will not be tried anymore, if the Rapid-Fail-Protection counter reaches a configurable limit within a configurable time limit. This prevents scenarios where worker processes hang or crash during startup.

Shutdown Limit

A worker process also has to shutdown in a configurable limit. If the shutdown doesn't happen in this time the worker process gets killed by WAS. This prevents resource overuse due to processes hanging in their shut-down phase. Additional shutdown settings allow an executable to be started (e.g. a debugger) when the shutdown doesn't complete within the allotted time.

CPU affinity

Configuration settings allow WAS to start worker processes that are affinitized to one or more CPUs. This prevents tenants from interfering with each other if they share the same physical machine.

User Profile

WAS can start worker processes with or without loading the user profile.

Security

Customizable User Account

IIS worker processes can run as a preconfigured process identity or built-in accounts (LocalService, LocalSystem, NetworkService). Built-in accounts are advantages because they don't require password management. If a custom user identity is used the password is automatically encrypted. Configuration settings can be replicated to multiple machines by sharing the configuration encryption keys across machines.

Job Object Features

Job objects allow administrators to restrict worker processes to a particular CPU limit. A configurable action is taken if this CPU limit is exceeded. Job objects will also make sure that processes spawned by the worker process get terminated.

Configuration Isolation and Security

Before WAS starts an Application Pool and its worker process it generates a unique configuration file for this Application Pool. WAS also creates a unique SID for each Application Pool (similar to Service SIDs introduced in Windows Server 2008). The Application Pool configuration file is then secured with this unique SID. This ensures that Application Pool configuration files can only be read by Administrators and the Application Pool itself.

Diagnostics and Monitoring

Event Logging

Events regarding invalid configuration, recycling, startup or shutdown of worker processes are reported to the System Eventlog.

Currently Executing Requests

WAS exposes a run-time and state control interface that allows scripts and tools to query for the currently executing requests of a particular worker process. This is useful to find requests that hang or requests that take a very long time to complete.

Performance Counters

All IIS performance counters get funneled through WAS. WAS gathers these performance counters because IIS counters are site-based and web applications can live in different Application Pools.

Recycling

Recycling allows the refresh of worker processes without losing a single request due to down-time. This is done via a feature called "overlapping recycling".

Overlapping Recycling

WAS does this by spawning up a new worker process parallel to the old one that is still handling requests. Once the new worker process is up it starts picking up requests from the request queue while the old worker process is instructed by WAS to stop picking up requests. Once the old worker process finishes all executing requests it shuts down. This feature is called "overlapping recycling". It ensures that no requests are lost during a recycle.

Recycling Configuration

Recycling parameters are configurable in the IIS configuration system.

Scheduled Recycling

Customers might want to recycle their applications based on a regular schedule. Via configuration settings recycling can be scheduled periodically, e.g. every 4 hours, every day at 1am etc.

Recycling Based on Memory Consumption

Applications might leak memory over time. WAS can monitor the memory consumption of each worker processes to ensure that no worker process uses more than its preconfigured limit. Reaching a configured virtual or private memory threshold will trigger the recycling of a worker process.

Recycling Based on Number of Requests

Recycling can also be configured based on the number of requests a particular worker process handled.

Custom Recycling

Custom code can custom health statistics and trigger a recycling via an API call to the WAS run-time and state API's.

Process Orphaning

Some errors only happen in a production environment. Killing worker processes ensures up-time but troubleshooting of these errors becomes difficult, e.g. if the failing worker process needs to be debugged. The process orphaning feature in WAS allows worker processes to be recycled without killing the failed worker process. Now a debugger can be attached to it. Additional process orphaning settings allow the execution of a process (e.g. a debugger) if orphaning happens.

Application Pool State Management

Application Pools can be stopped, recycled or started via publicly available API's, e.g. if an application has to be taken offline or if recycling has to be done based on parameters different from what's configurable in the applicationhost.config file.

Additional WAS Features

Load-Balancer Features

HTTP.SYS still listens on the network and will return a 500 HTTP error message if requests are not picked up by an Application Pool. This is a problem because for a Level 5 Load Balancers (TCP/IP) a 500 HTTP error looks like a valid TCP/IP connection. A WAS configuration setting can enable HTTP.SYS to reject connections instead of sending HTTP responses.

WAS can be configured to start worker processes with the following settings:

WoW64 Support

WAS can start 32-Bit or 64-Bit worker processes.

.NET Framework Preload

WAS can be configured to preload a particular version of the .NET Framework. This can make the troubleshooting of version conflicts much easier.

Web Gardens

A Web Garden is the term for an Application Pool that runs with multiple worker processes. Requests get distributed among these worker process instances using a round-robin mechanism.

WAS Multi Protocol Support

WAS not only host the HTTP stack. It can also host other protocols via its Listen Adapter and Worker Process Framework. WCF services take advantage of the WAS Multi-Protocol support. WCF protocols come with their own Listeners (e.g. the NET.TCP, NET.MSMQ or NET.PIPE Listener). These Listeners connect to WAS using the Listener Adapter Interfaces WAS provides.

Application protocols that take advantage of this infrastructure can host custom application code in the same .NET Application Domain as regular ASP.NET applications. They can also take advantage of the protocol-independent services the ASP.NET Hosting Environment provides, for example on-demand compilation, configuration support etc.

4. Summary

IIS7 and WAS offer a powerful process model for web-sites, web applications and web-services. It not only ensures availability but also ensures isolation and offers great troubleshooting and monitoring support.

IISRESET light

thomad — Tue, 06 May 2008 22:41:46 GMT

Note: This blog entry used to be on my old blog. I'm about to shut it down so I thought I replicate some of the content here.
Many IIS customers use IISRESET to get IIS back into a vanilla state. IISRESET is a pretty heavy hammer however and not needed most of the time - why would you restart FTP, WAS and W3SVC and all worker processes just because one of your web applications is locking a DLL or some content. Recycling the Application Pool causing the problem is usually enough. If you don't know which Application Pool is making the trouble you can recycle all of them. Here is how you do it with APPCMD:

Recycling the Default Application Pool:

%windir%\system32\inetsrv\appcmd recycle AppPool DefaultAppPool

You can use the APPCMD piping feature to recycle all Application Pools:

%windir%\system32\inetsrv\appcmd list apppools /xml | appcmd recycle apppools /in

Save this command as "%windir%\system32\IISPOOLRESET.BAT" and use it instead of using IISRESET.EXE. It probably takes some time to overcome the muscle memory of typing IISRESET<enter>.

In a Nut Shell: Shared Hosting Improvements on IIS7

thomad — Tue, 06 May 2008 07:43:00 GMT

We were meeting with AdHost today and I gave a quick 30-minute spiel on what we improved in IIS7 when it comes to Shared Hosting. Here is the list (not necessarily complete):

1) Shared Configuration

You can have the central IIS configuration file (applicationhost.config - the metabase is gone) on a central server and share it across multiple front ends. Not limited to Shared Hosting, but most Hosters love it. More info: http://learn.iis.net/page.aspx/264/shared-configuration/

2) Delegated Configuration

IIS configuration can be stored either centrally like in IIS 6.0 or a hoster can allow a customer to configure his own settings. No need to call up the hoster if default documents or authentication settings need to be changed. These settings can be delegated on a feature by feature basis or even on a per configuration property level. Delegated configuration settings are shared with ASP.NET in web.config files. More information: http://learn.iis.net/page.aspx/157/how-to-use-configuration-delegation-in-iis-7/

3) New Publishing Stack

IIS7 offers a brand-new publishing stack, i.e. FTP7 supporting SSL, WebDAV (map a drive in Explorer to connect to your web-site) and a new version of Frontpage Server Extensions.

More information:

FTP7: http://learn.iis.net/page.aspx/356/ftp-7-for-iis-70/
WebDAV: http://learn.iis.net/page.aspx/357/webdav-for-iis-70/
Frontpage Server Extensions: http://learn.iis.net/page.aspx/358/frontpage-server-extensions-fpse-for-iis-70/

4) Improved PHP Support

Wordpress, Drupal, phpBB are very popular and there are tons of other pre-packaged PHP applications out there. But PHP didn't run to well on the Windows platform. This changed with the FastCGI feature that was introduced for IIS 6.0 and 7.0. FastCGI is shipped as part of IIS in Windows Server 2008.

FastCGI/PHP: http://learn.iis.net/page.aspx/208/fastcgi-with-php/
MySQL: http://learn.iis.net/page.aspx/353/mysql-server/

And something else. The SQL Server drivers for PHP were slowing things down quite a bit. But not anymore. There is new PHP driver available written by the SQL Server team itself: http://www.microsoft.com/downloads/details.aspx?familyid=85f99a70-5df5-4558-991f-8aee8506833c&displaylang=en

5) Using Windows System Resource Manager for CPU Throttling

We are currently working on a new article that outlines how to use CPU Throttling for IIS Application Pools. WSRM allows a Hoster to make sure that no Application Pool monopolizes the CPU(s) of a particular machine. Here is a start: http://learn.iis.net/page.aspx/449/using-wsrm-to-manage-iis-70-apppool-cpu-utilization/

6) Directory Quota Support

Windows Server 2008 comes with a set of new file system features; one of them is the "File System Resource Manager" (FSRM) which supports directory quotas. With FSRM a Hoster can make sure the size of a directory doesn't get bigger than a configured limit. Soft limits generate an Eventlog entry or send e-mail, hard limits don't allow the files to be written at all. File Screens can ensure that certain file types and patterns are not uploaded. Hosters can also define individual templates that apply these settings in an uniform way. Directory quotas can be set via UI, via command-line tools or via scripting. More information: http://learn.iis.net/page.aspx/195/enabling-directory-quotas/

Configuration Example (only PowerShell for now):

param (    [string]$directory, 
    [int] $quotaLimit,
    [int] $threshold
);

$qMgr = new-object -com Fsrm.FsrmQuotaManager;
$quota = $qMgr.CreateQuota("$directory");
#quota in MB
$quota.QuotaLimit = ($quotaLimit*1024*1024);
$quota.AddThreshold($threshold);
#write to eventlog when $threshold is exceeded
$action = $quota.CreateThresholdAction($threshold, 1);
$action.EventType = 1;
$action.MessageText = "Disk quota ($quotaLimit MB) for directory $directory exceeded.";
$quota.Commit();

7) Bandwidth Throttling

Media content like videos or audio files (e.g. Podcasts) become more and more pervasive in Hosted Environments. But Media content can become quite a problem. Usually Hosters want to provide fast download times to maximize the user experience. This is not the case with media content however and here is why:

If you look at the grand scheme of things only a fraction (about 20%) of media content is consumed before the user browses to something else. Just ask yourself how many youtube videos you really watch from start to finish.

If you have a fast connection 100% of the media file gets downloaded until the user stops watching. 80% of the bits transferred were wasted. That's not a problem if you have unlimited bandwidth, but who has that? The IIS7 Bitrate Throttling Module allows Hosters to progressively throttle the bandwidth of media and other content to a desired rate or to the rate that is encoded in the media file itself. More information: http://learn.iis.net/page.aspx/147/bit-rate-throttling-setup-walkthrough/

8) IIS7 Core Web Server Improvements

a) No need for an anonymous account anymore.

Whenever requests are not authenticated IIS uses an anonymous user account to access web content. In IIS6 a default account is created named "IUSR_<machinename>. For isolation purposes Hosters usually create one anonymous user account for each web-site.

You don't have to do that anymore in IIS7. By leaving the anonymous username and password empty IIS7 Application Pools will use the Application Pool identity itself to access anonymous content. So if a Hoster has an individual Application Pool account for each customer he doesn't have to create a second anonymous account on top of that. The single Application Pool account is enough.

Configuration Example (removes the anonymous user server-wide):

windir%\system32\inetsrv\appcmd set config -section:anonymousAuthentication 
-username:"" -password:""

User Interface

b) dynamicIdleThreshold

In most Shared Hosting environments very active (hot) sites live together with infrequently requested (cold) sites. When too many Application Pools for these sites get spawned up memory becomes sparse and the Windows Memory Manager starts moving infrequently used memory pages to the pagefile. If memory pressure gets too high memory is over-committed and the machine starts thrashing. In this case the system excessively swaps memory pages between RAM and pagefile because more and more of the required memory pages reside in the page file and need to be paged back into memory. Excessive swapping of memory pages is also called thrashing. It usually starts happening when about 130% of the memory is committed. Example: For a machine with 2GB physical RAM thrashing might start at 2.6GB (2GB RAM + 600MB in the pagefile = 2.6GB/2.0GB = 130%).

Thrashing is not fatal, but it certainly slows down the system. Requests start to take longer than usual. The worse thrashing gets the longer responses will take.

Now back to IIS: there is a remedy in IIS 7.0 when memory becomes sparse. It's called dynamicIdleThreshold. All you have to do is to configure the percentage of memory committed, e.g. 130. Should the system get close to this limit IIS will start reducing the idle timeout of Application Pools. IIS7 starts

At 80% of the dynamicIdleThreshold setting it reduces the idle timeout of all Application Pools by half, e.g. the 20 minute default timeout would be set to 10 minutes.
At 85% of the dynamicIdleThreshold setting it reduces the idle timeout of all Application Pools to 1/4th, the 20 minute default timeout would be set to 5 minutes.
At 90% of the dynamicIdleThreshold setting it reduces the idle timeout of all Application Pools to 1/8th, i.e. 2.5 minutes if you use the 20 minute default.
At 95% of the dynamicIdleThreshold setting it reduces the idle timeout of all Application Pools to 1/16th, i.e. 1 minutes if you use the 20 minute default.
At 100% of the dynamicIdleThreshold setting it reduces the idle timeout of all Application Pools to 1/32th, i.e. 1 minutes if you use the 20 minute default (one minute is the minimum).

For math wizards: Let's suppose we set dynamicIdleThreshold to 130 on our machine with 2GB physical RAM. The 80% threshold would be reached if 2.08GB of the memory are committed (80% of 2.6GB (130% dynamicIdleThreshold)).

As a net result cold Application Pools get timed out much faster while hot Application Pools don't get timed out because they continuously receive requests and their idle timeout counter gets reset constantly. An Eventlog entry is generated every time an Application Pool is timed-out due to a reduced time-out. In our test environment we can host up to 10 times more Application Pools when enabling the dynamicIdleTimeout setting.

The side effect of this setting is of course that process data like in memory session state gets deleted if an Application Pool times out.

Here is how you configure the dynamicIdleThreshold setting:

%windir%\system32\inetsrv\appcmd set config 
-section:system.applicationHost/webLimits -dynamicIdleThreshold:130

c) Static and Dynamic Compression

Compressing responses can be a big benefit for HTTP clients which have limited bandwidth, for example cell phones or dial-up connections. It can also save a lot of bandwidth.

IIS automatically compresses static content like html or text files. To do this the HTTP client has to send the "Accept-Encoding: gzip" header. Most HTTP clients like Internet Explorer or FireFox do this automatically. The big advantage with static compression is that IIS can cache the compressed representation of the content in memory and even on disk. For the next response the already compressed representation of the content gets sent. This way it doesn't have to recompress for every new request. Compressing most static content is the default for IIS7.

Dynamic content (e.g. ASP.NET, PHP or other Web Application Frameworks) is a bit trickier because the compression has to be done for every response and that costs CPU cycles of course. Due to the benefits described above it might be a good idea to turn dynamic compression on if spare CPU cycles are available however. This is not an all-or-nothing either because IIS7 has some high-watermark - low-watermark settings that allow to disable compression if the CPU reaches a certain limit, e.g. 90%.

Configuration Examples:

Enable Dynamic Compression

%windir%\system32\inetsrv\appcmd.exe set config "Default Web Site" 
-section:system.webServer/urlCompression /doDynamicCompression:"True"

Set Dynamic Compression High Watermark

%windir%\system32\inetsrv\appcmd.exe set config  
-section:system.webServer/httpCompression /dynamicCompressionDisableCpuUsage:"80"

d) Application Pool SIDs

IIS7 generates a unique Security Identifier (SID) for every Application Pool it starts. This allows to isolate content for two or more Application Pools even if they run as the same account. You can ACL resources like files and directories for these individual Application Pool identities. See Ken Schaefer's blog for more details: http://adopenstatic.com/cs/blogs/ken/archive/2008/01/29/15759.aspx

Configuration Example:

icacls C:\content\site1\test.txt /grant "IIS APPPOOL\Site1AppPool":R

e) Failed Request Tracing

Troubleshooting Web Server errors is hard. Especially in Hosting Environments where customers write the code but the Hoster has to troubleshoot. Often times there is a lot of code that gets executed before a response is returned. And a request can fail anywhere in the request processing. Failed Request Tracing allows an IIS administrator to specify rules for failed requests. If a request conforms to the rule (e.g. request takes longer than 30 seconds) IIS will write a detailed trace of the "failed" request to disk. Here is an example of the output:

More information on Failed Request Tracing: http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis7/

f) Running Wow64

RAM is cheap and having machines with more than 4GB of RAM is not outrageous anymore. To address more than 4GB of RAM a 64-Bit Operating System is necessary. Windows Server 2008/64-Bit is thoroughly tested and I would recommend running on 64 instead of 32-Bit. Compatibility is important however and IIS software like PHP or ISAPI filters and extension are not be available as 64-Bit binaries. 32-Bit is also quite a bit lighter-weight and doesn't need as much memory. The solution: install the 64-Bit version of Windows Server 2008 but run 32-Bit Application Pools. The "enable32BitAppOnWin64" is a per Application Pool setting, i.e. 32-Bit and 64-Bit Application Pools can be run side-by-side.

Configuration Example:

appcmd apppool set /apppool.name:My32BitAppPool /enable32BitAppOnWin64:true

User Interface

Summary

There are a ton of new features in Windows Server 2008 and IIS7 that make Shared Hosting easier. I tried to outline the most important ones but I probably forgot a bunch. Stay tuned for future updates.

Understanding IIS7 Request Restrictions on Windows Vista

thomad — Thu, 01 May 2008 22:31:00 GMT

Note: This blog entry used to be on my old blog. I'm about to shut it down so I thought I replicate some of the content here.

You might have run into the IIS connection limit on Windows XP. IIS 5.1 on Windows XP allowed 10 concurrent connections before it displayed a 403.9 error message (Access Forbidden: Too many users are connected). Not too user friendly.

The only restrictions we have on Windows Vista is the number of concurrent requests IIS will execute. This is fundamentally different from the Windows XP connection limit. By limiting concurrent requests you will probably never see an error message. Request that cannot be handled because the concurrent request limit is reached will still be queued.

There is only a problem if the request queue overflows - a "500 Server Busy" error. The request queue is pretty big though. Thousands of requests by default. The net effect is that the execution of too many concurrent requests will not generate an error but it might slow your server down for a little bit. Windows Server 2008 doesn't have any request restrictions.

Here is how many concurrent requests IIS7 allows on Windows Vista SKUs:

Windows Vista Home Basic*	3
Windows Vista Home Premium	3
Windows Vista Ultimate	10
Windows Vista Professional	10
All Windows Server 2008 SKUs	unlimited

*Home Basic contains the IIS process activation and HTTP processing infrastructure, but it cannot be used as a traditional web server. It also doesn’t include the IIS administration tool.

IIS7 PowerShell Provider Podcast

thomad — Tue, 29 Apr 2008 22:43:53 GMT

Saw a post on forums.iis.net last week about a Podcast on our new IIS7 PowerShell Provider. Being the Program Manager for this thing I thought maybe these guys want me to participate in the Podcast. And sure enough - Jonathan and Hal were interested. And here it is the podcast link and the interview topics:

http://powerscripting.wordpress.com/2008/04/26/powerscripting-podcast-episode-23-iis7-special/

Interview Topics

Who are you
What’s your background at MS and elsewhere
Talk about the IIS7 management cmdlets
- Get/Set-WebConfiguration
- Start-WebItem
- Remove-WebConfigurationProperty
- Ability to use XPath filters
Talk about the IIS7 PSprovider
- Provider timeline - 2nd beta in June, final in October
- Features
  - ability to configure IIS and ASP.net, sites, vdirs, apps, all that
  - ability to delegate
  - root of namespace: sites, app pools
What does the future hold (that you can discuss)
- We talk about Server Core

Cheesy Web Server Performance Test With PowerShell

thomad — Tue, 22 Apr 2008 07:35:00 GMT

Instead of writing WCAT scripts I wrote my own little perf test client. The following .PS1 script works pretty well to do some very basic performance testing. Just enter the URL and the duration in seconds, e.g. ".\webperf.ps1 http://localhost/ 15".

The script instantiates the Net.WebClient class and calls the DownloadString member to make a HTTP request. The response is redirected to $null. The script checks on every request if the duration is expired. It reports every 100 successful responses.

param

    [string]$url,

    [int]$duration

"URL: $url";

"Duration: $duration seconds`n";

$starttime = get-date;

$i=0;

$webClient = new-object Net.WebClient;

while (1)

    $webClient.DownloadString($url)>$null;

      $i++;

      $timespan = new-timespan $starttime;

      if ($timespan.TotalSeconds -ge $duration)

           "`n`n$i requests for url $url served in $duration seconds."

           break;

      else

           if ($i%100 -eq 0)

                $dursec = [int]$timespan.TotalSeconds;

                write-host -noNewLine "`r$i requests served ($dursec seconds)";

Requesting the IIS default page I am able to get around 48000 requests per minute (800 per second) on my Lenovo T61p on Vista Ultimate SP1. How many can you do?

IIS 7.0 PowerShell Provider Tech Preview 1

thomad — Mon, 14 Apr 2008 20:10:00 GMT

Finally, IIS 7.0 has a PowerShell Provider!

The IIS7 PowerShell Provider allows you to

Create Web-Sites, Web Applications, Virtual Directories and Application Pools
Change Simple Configuration Properties on Web-Sites, Application Pools, Web Applications and Virtual Directories
Add and Change Complex Configuration Settings
Query Run-time Data (Web-Site State, Application Pool State, Currently Executing Requests)
Execute Advanced Configuration Tasks, Scripting, Integration with other PowerShell Snap-Ins and features
Search and Discover Configuration Settings

Here is a screen shot of how to create a new IIS app:

DOWNLOAD:
Tech Preview 1 of the IIS 7.0 PowerShell Provider can be found here:
x86: http://www.iis.net/downloads/1664/ItemPermaLink.ashx
x64: http://www.iis.net/downloads/1665/ItemPermaLink.ashx

FORUMS:
Go to our PowerShell forum if you need support or if you are looking for 'Tips and Tricks'
http://forums.iis.net/1151.aspx

WALKTHROUGHS:
We have 9 walkthroughs for you to get familiar with the IIS 7.0 PowerShell Provider:
http://learn.iis.net/page.aspx/447/managing-iis-with-the-iis-70-powershell-provider/

Have fun!