Mark Arend

Host Named Site Collections (HNSC) for SharePoint 2010 Architects

Mark Arend — Wed, 30 May 2012 23:20:07 GMT

I recently came across a very good blog posting on Host Named Site Collections (HNSC), and I recommend it: What Every SharePoint Admin Needs to Know about Host Named Site Collections by Kirk Evans. Kirk has lots of good details here that will help answer questions on this popular feature with some obscure underpinnings.

However, I worked up other material on HNSC some time ago that’s not found in Kirk’s article. Mine is targeted more to the SharePoint Architect who is considering the best way to design and use HNSC in a SharePoint farm.

One small discrepancy with Kirk’s article—he relates a commonly accepted fact of HNSC: “if you are going to implement host named site collections, you should not use a host header for the containing web application that houses the host named site collections or it will not work properly.” The most unfortunate thing about this is that it requires you to use different IP addresses or ports if you want more than one web application to host HNSC. Fortunately, I was able to find that Microsoft does indeed support Host Headers and HNSC on the same web application; I give the procedure below and tips for verifying support through Microsoft Premier. This provides an additional degree of flexibility for architecting your SharePoint farm.

Host Named Site Collections

The mechanism for consolidating existing web applications into individual site collections, yet retain their existing URLs, is to use Host-Named Site Collections (HNSC). This is a feature of SharePoint that allows individual site collections to have their own top-level URL. This section explores pertinent points from TechNet article Plan for host-named site collections (SharePoint Server 2010).

Benefits

Much more scalable for vanity URLs because there can be thousands of site collections per web application, but only 20 web applications per farm.
Managed paths may still be used on an HNSC web application, so that the more familiar “path-based” site collections may be created.

An explicit or wildcard managed path may be created in the “normal” way, so that non-HNSC site collections may be created on them, as usual. This is done using Central Admin.
Also, explicit and wildcard managed paths may be created on the web app that will apply to each HNSC site collection created. That is, each HNSC site collection will have the same managed paths that are created this way. This is done using PowerShell scripts; an example below shows adding these kind of managed paths..

Drawbacks

HNSC may only be created using PowerShell scripts (or programmatically), not via the SharePoint UI.

This introduces additional effort in creating them, similar to the effort of creating an additional web application, which requires a farm administrator, but with the added requirement of having to execute a PowerShell script on one of the farm servers. This also means that a higher level of access—server administrator—is required to create new HNSC. It would be possible to develop a simple extension to SharePoint that could be executed from Central Administration (or other administration site) to call the appropriate PowerShell script, so that this operation could be delegated back to a farm administrator.

HNSC cannot use alternate access mappings. Host-named site collections are automatically considered to be in the Default zone, and the URL of the request must not be modified between the end user and the server.

Special considerations

For using SSL and SSL with encryption offloading, some special considerations apply.

Because SharePoint Server 2010 uses the public URL in the Default zone of the Web application to determine whether host-named site collections will be rendered as HTTP or SSL, you can now use host-named site collections with off-box SSL termination. There are 3 requirements to use SSL termination with host-named site collections:

The public URL in the Default zone of the Web application must be an HTTPS-based URL.
The SSL terminator or reverse proxy must preserve the original HTTP host header from the client.
If the client SSL request is sent to the default SSL port (443), then the SSL terminator or reverse proxy must forward the decrypted HTTP request to the front-end Web server on the default HTTP port (80). If the client SSL request is sent to a non-default SSL port, then the SSL terminator or reverse proxy must forward the decrypted HTTP request to the front-end Web server on the same non-default port.

To use host-named site collections with off-box SSL termination, configure your Web application as you normally would for SSL termination and ensure that it meets the requirements described above. In this scenario, SharePoint Server 2010 will render links of its host-named site collections in that Web application using HTTPS instead HTTP.
SharePoint Server 2010 does not support a host-named site collection using both HTTP- and SSL-based URLs simultaneously. If some host-named site collections need to be available over HTTP while other host-named site collections need to be available over SSL, separate the host-named site collections into two different Web applications dedicated for that type of access. In this scenario, HTTP host-named site collections should be in a Web application dedicated for HTTP access and SSL host-named site collections should be in a Web application dedicated for SSL access.

Both types of site collections may be mixed on the same web application. Comparison of features:

Feature	Path-Based Site Collection	Host-Named Site Collection
Managed Paths	YES	YES*
Create new site collection at top level	SharePoint UI	PowerShell Script
Create new site collection on a managed path	SharePoint UI	PowerShell Script
Authentication Zone	Choose: Default, Internet, Intranet, Extranet, Custom	Default only
Alternate Access Mapping (AAM)	YES	No
SSL	YES	YES
SSL Termination (encryption offloading)	YES	YES**
Kerberos	YES	YES

* Managed paths for HNSC and the site collections created on them must both be created using PowerShell scripts. The example later in this section shows both of these techniques.

** Offloading encryption for SSL sites is possible with HNSC, but has slightly more stringent requirements for configuration. Specifically: the public URL in the Default zone of the Web application must be an HTTPS-based URL, and the SSL terminator or reverse proxy must preserve the original HTTP host header from the client. Also, best to use standard ports 80 and 443.

Kerberos authentication works with HNSC as it does with an HH web application—must register a Service Principal Name (SPN) for the external URL on the identity of the application pool that hosts the web app.

The strategy recommended is:

Create a Host-Named Site Collection for every request for a site that requires its own “vanity” URL. That these URLs can be either of the format http://newsite.customer.com, or even http://www.newsite.com, as far as SharePoint is concerned.

DNS must be configured to serve the appropriate addresses. If there are more than one web application in the farm that contain Host-Named Site Collections, then it will probably not be possible to use a wildcard DNS “A” record such as *.customer.com to direct all traffic correctly, but each HNSC will need a separate DNS “A” record. See analysis of options for creating multiple web apps, below.
This option is for a less frequent, more “long-term” type of configuration request, justifying the additional effort.

Create a Path-Based Site Collection for every request for a site that falls in the category of an existing web application. These use URLs such as http://teams.customer.com/sites/TeamOne, where TeamOne is the path-based site collection that may be created through the SharePoint UI. Explicit managed paths may also be used, such as http://teams.customer.com/TeamTwo.

This option is for a more frequent type, “immediate” type of configuration request for new teams, communities, projects, etc., and can be accomplished entirely within SharePoint without server or external dependencies.

From an organization standpoint, it may be desirable to use a few individual web applications to organize sites by certain criteria, to better manage different types of sites from a standpoint of security, reliability, performance, scalability or SLA. For the examples below, we’ll consider two web applications labeled “Standard” and “Sensitive”, then deploy HNSC to each web application.

Basic instructions for creating HNSC assume there is only one web application that supports these on the farm. When deploying multiple web applications that each will host HNSC, some additional complexity is required to create the web applications correctly. The following table compares the options for accomplishing this.

Options for creating multiple web apps for Host-Named Site Collections (HNSC)

When creating multiple web applications in SharePoint, they are often configured to use different Host Header (HH) names to ensure uniqueness, while having the same IP address and port (80 or 443). Host Header names on web applications are largely incompatible with Host-Named Site Collections, but can be made to work in a supported way. The following table examines each option for creating multiple web apps, and their implications for HNSC:

Option	Implications	Supported?
1. Create web apps using different ports	Configure entirely through SharePoint UI. Some HNSC will require nonstandard ports in URLs; e.g. http://teams.customer.com:81 All URLs must include port numbers (That is, one web application hosting HNSC could use default port 80, but any additional web apps hosting HNSC would need to use a different port, and this would be required in all URLs hosted by those web applications.)	Yes (out-of-box)
2. Create web apps using different IP addresses	Requires multiple IP addresses, one per each web app that’s hosting HNSC. Must manually configure different IP addresses, per web app, per server. Must create VIP for each HNSC web app Must configure DNS for each HNSC to appropriate VIP Gives desired URLs, consolidated into multiple web applications, separated by category to enforce security, performance, reliability and scalability. However, there is additional complexity of maintaining different IP addresses and mappings in DNS and the load balancer.	Yes (KB927376, applies also to SharePoint 2010)
3. Create web apps using different host headers	Blocks HNSC URLs being recognized by IIS. Must manually configure IIS to bind HNSC URLs to unblock, per web app, per server. This option has all the benefits of option 2, with the added benefit that DNS can be configured with wildcard third-level domain names so that creation of a new HNSC does not necessarily require a new DNS record.	Yes (reference Microsoft Premier case 111022261144490)

Option 1 – Identify by Port

This is the simplest approach, described in the reference article Plan for host-named site collections (SharePoint Server 2010).

The problem with this approach is that if you want to configure more than one web application for HNSC, then only one of them may use default port 80 or 443, while the others must use different ports; many organizations don’t wish to require their users to enter ports for URLs to the various sites they plan for their intranet.

Option 2 – Identify by IP Address

This approach allows the creation of multiple web applications that are uniquely identified by IP address. This means that “clean” URLs may be presented to the end user, without ports. The drawbacks here are not necessarily terrible, but worth considering:

Multiple IP addresses are required on the farm WFE servers; one per HNSC web application per WFE server. This isn’t hard to configure, but some companies have policies against it that must be waived with a business justification.
Creating a new HNSC web application means a new IP address must be provisioned onto each of the WFE servers, and into the load balancer, with a new Virtual IP (VIP) address as well.

IP Addresses. The first web application configured to host HNSC could use the “default” IP address of the server, but each additional web application to host HNSC must have a unique IP address. To accomplish this, the Network Interface Card (NIC) providing user access on each WFE server must be configured with multiple IP addresses. Naturally IP addresses will be different across all servers.

Load Balancer. The load balancer that distributes requests across all web front end servers must be configured with multiple Virtual IP (VIP) addresses. Each VIP will load balance out to the IP addresses associated with a separate web application across the WFE servers.

DNS. Finally, DNS must be configured with the root URLs of the HNSC, each targeted to the appropriate load-balanced VIP. A wildcard “A” record like *.customer.com cannot be used, unless DNS can be configured to guarantee that it will first honor all explicit “A” records such as teams.customer.com.

The following diagram shows conceptually the relationship among these aspects, also illustrating how non-HNSC web applications are configured:

In this diagram, each WFE has a “default” IP address; this is used for all requests to the server that are not host-named site collections, including Host Header web applications (e.g. mysite.customer.com, app.customer.com) and web applications distinguished with a separate port. Also, one web app hosting HNSC may also use this IP address. Then, additional IP addresses are shown, corresponding to each additional web application serving HNSC. Note, this arrangement minimizes the number of IP addresses required on each server. If that is not of great concern, it will be clearer to simply use a unique IP address for each HNSC web app, not configuring one of these on the default IP.

There must be a load-balanced VIP corresponding to each of the separate IP addresses across the WFE servers. VIP.1 is the “typical” VIP that distributes inbound requests to each server’s default IP address. VIP.2 distributes inbound requests to each server’s next additional IP address, and so on.

Note: for DNS entry *.customer.com, this will only work if DNS can be configured to examine inbound requests against all other explicit A records prior to following this wildcard entry.

How to create web applications with different IP addresses for HNSC

The following PowerShell scripts illustrate steps for configuring different IP addresses per web app, per server. This script creates two web apps, configures them to different IP addresses, then creates managed paths and a series of host-named site collections on the web apps. This approach follows KB article 927376, which gives the supported method for configuring web applications on different IP addresses. Written for MOSS 2007, KB article 927376 also applies to SharePoint 2010.

Full example scenario for option 2

1. Create web applications on the farm using host header names. Use the SharePoint UI to do this, or a PowerShell script such as the following.

# Create web apps using Host Headers so multiple can be created on port 80.

# Run this once per farm

New-SPWebApplication -ApplicationPool "SharePoint Web Apps" -Name "Portal" –HostHeader Portal -DataBaseName "WSS_Content_Portal" -DatabaseServer "FARMSQL" -Path "C:\inetpub\wwwroot\wss\VirtualDirectories\Portal" -Port 80 -URL "http://Portal:80"

New-SPWebApplication -ApplicationPool "SharePoint Web Apps" -Name "Teams" -HostHeader Teams -DataBaseName "WSS_Content_Teams" -DatabaseServer "FARMSQL" -Path "C:\inetpub\wwwroot\wss\VirtualDirectories\Teams" -Port 80 -URL "http://Teams:80"

2. Change the bindings on the web applications just created in order to use IP addresses. It’s recommended to do this using PowerShell because the scripts can be created ahead of time and reviewed for correctness over multiple runs, instead of relying on a manual configuration via IIS manager each time. Note that the IP addresses will be different for each WFE server.

# Change bindings on web applications from host headers to IP Addresses

# Run this once on each WFE server

# Requires IIS Administration module

Import-Module WebAdministration

# Remove bindings with Host Headers from IIS sites

Remove-WebBinding -Name "Portal" -HostHeader "Portal"

Remove-WebBinding -Name "Teams" -HostHeader "Teams"

# Replace with new bindings having unique IP Addresses

New-WebBinding -Name "Portal" -IPAddress 192.168.3.11 -Port 80

New-WebBinding -Name "Teams" -IPAddress 192.168.3.12 -Port 80

3. Create Host-Named Site Collections on the web applications. PowerShell must be used to do this because the SharePoint UI does not provide this specific configuration option.

# Create Host-Named Site Collections (HNSC)

# Run this once per farm

# Add managed paths (for all HNSC in farm)

New-SPManagedPath "it" -Explicit -HostHeader -ErrorAction Continue

New-SPManagedPath "sec" -Explicit -HostHeader -ErrorAction Continue

# Create HNSC on Portal

New-SPSite http://communities.customer.com -HostHeaderWebApplication http://Portal -Name "CUSTOMER Community Sites" -Template "STS#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://intranet.customer.com -HostHeaderWebApplication http://Portal -Name "The Intranet" -Template "BLANKINTERNETCONTAINER#0" -OwnerAlias CUSTOMER\Administrator

# Create HNSC on Teams, with sites on managed paths

New-SPSite http://teams.customer.com -HostHeaderWebApplication http://Teams -Name "CUSTOMER Team sites" -Template "STS#1" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://teams.customer.com/it -HostHeaderWebApplication http://Teams -Name "CUSTOMER IT - Information Technology" -Template "BLOG#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://teams.customer.com/sec -HostHeaderWebApplication http://Teams -Name "CUSTOMER SEC - Information Security" -Template "BLOG#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://project.customer.com -HostHeaderWebApplication http://Teams -Name "Project Tracking" -Template "SGS#0" -OwnerAlias CUSTOMER\Administrator

Option 3 – Identify by Host Header

The TechNet article we’ve been referencing says several things against this option:

“In most cases, applying a host header binding at the IIS Web site level makes it impossible to access host-named site collections through the IIS Web site. This is because IIS will not respond to requests for host names that differ from the host header binding.”

“To ensure that both types of site collections are accessible to users, do not put host header bindings on the IIS Web site assigned to the Default zone of your Web application, if you have host-named site collections in that Web application.”

“You can manually modify host header bindings on the IIS Web site from the IIS Manager, but this is not recommended. Any changes you make using the IIS Manager will not be recorded in SharePoint Server 2010. If SharePoint Server 2010 tries to provision an IIS Web site on another computer in the farm for the same Web application and zone, the original host header binding is used instead of the modified binding.”

However, these can be overcome by following a process to maintain host header bindings on the IIS web site. A proposal to use this approach has been approved by Microsoft Premier as supportable; reference Microsoft Premier case “[REG:111022261144490] prem/SharePoint Server 2010 AL/Requesting RCS for support ability scenario”

The main implications of this option are the operational procedures for creating a new HNSC. Instead of using the SharePoint Central Administration console, or the feature for self-service site creation, PowerShell commands must be run, in the following specific order:

1. On each WFE server in the farm, must add a host header binding to the IIS web site hosting the web application/zone. This could be done via IIS manager, but it’s recommended to use a PowerShell script for accuracy and repeatability. Example of adding the HNSC URL binding for “sitename.customer.com” to the web application whose host header name is “Standard”:

New-WebBinding -Name "Standard" -HostHeader "sitename.customer.com"

(This requires the IIS Administration Module to be imported for PowerShell)

2. Once for the whole farm, must create the HNSC using a PowerShell command; it can’t be done via the SharePoint UI. Example of creating the HNSC “sitename.customer.com”

New-SPSite http://sitename.customer.com -HostHeaderWebApplication http://Standard -Name "Our Work Site" -Template "STS#1" -OwnerAlias CUSTOMER\Administrator

The steps must be performed in this order so that IIS is configured to respond correctly to the desired HNSC URL before SharePoint is asked to create it.

Note: if the -Template argument is omitted from the New-SPSite command, then the first access of the site collection will prompt for a choice of site template to be instantiated.

Note: this additional operational procedure would be necessary for option 2 also, because that is the recommended technique for creating web applications on different IP addresses.

Full example scenario for option 3

The PowerShell scripts below are an example showing how a set of web applications and site collections can be created using host headers to uniquely identify the web applications, thus avoiding the drawbacks of the “port” and “ip address” approaches.

Sample steps only

Web app “Standard” will host intranet.customer.com and communities.customer.com
Web app “Sensitive” will host teams.customer.com and projects.customer.com. In addition, managed paths are added so that teams will also contain teams.customer.com/sec and teams.customer.com/it.

1. Create web applications on the farm using host header names. Use the SharePoint UI to do this, or a PowerShell script such as the following. Scripting is recommended for repeatability (dev, test and prod!)

# Create web apps using Host Headers so multiple can be created on port 80.

# Run this once per farm

New-SPWebApplication -ApplicationPool "SharePoint Web Apps" -Name "Standard" –HostHeader Standard -DataBaseName "WSS_Content_Standard" -DatabaseServer "FARMSQL" -Path "C:\inetpub\wwwroot\wss\VirtualDirectories\Standard" -Port 80 -URL "http://Standard:80"

New-SPWebApplication -ApplicationPool "SharePoint Web Apps" -Name "Sensitive" –HostHeader Sensitive -DataBaseName "WSS_Content_Sensitive" -DatabaseServer "FARMSQL" -Path "C:\inetpub\wwwroot\wss\VirtualDirectories\Sensitive" -Port 80 -URL "http://Sensitive:80"

2. Add new bindings on web applications so IIS recognizes Host-Named Site Collection names

# Add HNSC bindings on web applications

# Run this once on each WFE server

# Requires IIS Administration module

Import-Module WebAdministration

# Add new bindings for HNSC

New-WebBinding -Name "Standard" -HostHeader "intranet.customer.com"

New-WebBinding -Name "Standard" -HostHeader "communities.customer.com"

New-WebBinding -Name "Sensitive" -HostHeader "teams.customer.com"

New-WebBinding -Name "Sensitive" -HostHeader "projects.customer.com"

3. Create Host-Named Site Collections on the web applications.

# Create Host-Named Site Collections (HNSC)

# Run this once per farm

# Add managed paths (for all HNSC in farm)

New-SPManagedPath "it" -Explicit -HostHeader -ErrorAction Continue

New-SPManagedPath "sec" -Explicit -HostHeader -ErrorAction Continue

# Create HNSC on Standard

New-SPSite http://intranet.customer.com -HostHeaderWebApplication http://Standard -Name "The Intranet" -Template "BLANKINTERNETCONTAINER#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://communities.customer.com -HostHeaderWebApplication http://Standard -Name "Community Sites" -Template "STS#0" -OwnerAlias CUSTOMER\Administrator

# Create HNSC on Sensitive, with sites on managed paths

New-SPSite http://teams.customer.com -HostHeaderWebApplication http://Sensitive -Name "Team sites" -Template "STS#1" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://teams.customer.com/it -HostHeaderWebApplication http://Sensitive -Name "IT - Information Technology" -Template "BLOG#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://teams.customer.com/sec -HostHeaderWebApplication http://Sensitive -Name "HR – Human Resources" -Template "BLOG#0" -OwnerAlias CUSTOMER\Administrator

New-SPSite http://projects.customer.com -HostHeaderWebApplication http://Sensitive -Name " Project sites" -Template "SGS#0" -OwnerAlias CUSTOMER\Administrator

Note: Wildcard managed paths for HNSC can also be added, by omitting the -Explicit parameter from New_SPManagedPath.

This type of configuration is supported, and illustrates a way to partially automate this configuration on each WFE in the farm.

Measuring Concurrent Site Users – SharePoint 2010 and other

Mark Arend — Sat, 25 Feb 2012 01:35:25 GMT

This post was written as I was working to analyze traffic on SharePoint 2010, but it could be used for any website log traffic analysis.

Measuring concurrent users on your web site over time is important to understand characteristics of the site’s performance—its overall throughput—based on its load. In fact, planning worksheets on TechNet tell us to find a number for concurrent users, but they don’t tell us how to get that number.

However, the idea of concurrent users is hard to pin down. What exactly does that mean? Conceptually, you might think of it as the number of users who are hitting the site right now. Ok, but what does right now mean? We really must define it as a specific duration of time, but how long? I’ve seen many calculations count the number of unique users every hour. Is that meaningful? I think in many cases, it’s not.

Let’s assume for the moment that your web site doesn’t use session state, it just gets requests and returns results. For this, it’s got to do some work to assemble the result. SharePoint especially has a lot of construction to do, gathering data from the database, the user profile, the file system, applying master pages, page layouts, maybe executing page controls and web parts, phew! So this is going to take some time on the server, and if another user sends a request, then those tasks will add to the overall load. But if that doesn’t happen until later in the same hour, then those tasks don’t really add to the overall load. So I think the duration should be less than an hour. Note, if your site does use session state, that changes things slightly… more later.

How short should the duration be?

Let’s look at the impact of measuring logs using different durations. I used LogParser to query through some logs from a small SharePoint 2010 farm; 8 million requests over a 2 week period, collated and quantized into behavior per hour of day. This graph counts the number of distinct (unique) IP addresses that were found in four different-sized “buckets of time” which I’m calling quanta. It does not show the number of requests. For 5 minute quanta, I count the total number of unique IP addresses in each 5 minute period throughout the day, then average those results over each hour. For 60 minute quanta, I count the total number of unique IP addresses in that whole hour.

Wow, that’s quite a difference. If we quantize users into 5 minute buckets versus 60-minute buckets, we find a wide range of results. Designing a production infrastructure for SharePoint, I’m certainly going to want to know if it has to support 20 or 120 concurrent users during those busy parts of the day.

Abstractly, I think the size of the quanta into which we count distinct users hitting the farm should be somewhat longer than it would take the server to “come to rest” after a heavy stream of incoming requests suddenly stopped, allowing all requests to be finished and results returned. It seems to me that after this time, since the server is done processing those requests, it can take more without additional load. Am I going to try and compute this time? Erm, no. I’m going to WAG it at about 5 minutes in general, based on some undocumented experiences over many years. But maybe the server could only finish all of its work in that time because nothing else was coming in. Ok, so let’s double it: 10 minutes.

Of course there’s a body of knowledge around load testing that might help us here. One site that has a lot of great detailed technical information about load testing is the blog at webperformance.com. For load testing, ramp-up is more important than the ramp-down abstraction above. They like to ramp-up over a long period—many minutes to several hours—in order to understand the behavior of the system under different loads. But ramp-down is more about leaving the system in a stable state, allowing all user requests to complete normally, instead of just killing all virtual user threads in-request. In this case, the ramp-down is dependent mostly on maximum request processing time, which is pretty much page latency. Now, we may want our page latency to be just a few seconds in most cases, allowing more on pages that do “complex” things like generate reports. But even with relatively fast pages, ramp-down takes a while longer. Following this line of thinking, the 5 minute and 10 minute durations for concurrency continue to seem more reasonable than larger buckets.

For another sanity check. How do people typically use SharePoint and other publishing/collaboration intranet sites? Again from personal experience, I tend to use individual sites in relatively short bursts. If I’m searching for information, I’ll want some results in just a few minutes, or I’m going to give up and try something different. Or if I’m collaborating, I’ll be uploading and downloading documents one at a time or a few at a time, then going off to work with those documents. So, I might have multiple SharePoint sites open all day long, but my direct interactions with them are bunched together, separated by other work. For me, the idea of 10 minutes well encompasses each of my individual direct interactions with collaborative sites such as SharePoint.

I’m really talking myself into this figure of 10 minutes as a guideline for measuring concurrency. And I welcome any debate on the topic… please post comments!

So, what if you do use session state on your site? For SharePoint, the typical session state available for developers is disabled by default (see my Session State blog posting for more information). But if you enable it, or are using another web site host with session state, then whenever a user sends an authenticated request to the server, it will set aside some resources to maintain that session, for some period of time. Way back in my ASP days, I recall 20 minutes being a common default, and it’s always configurable. If session state is maintained in the database, that’s fewer resources that the WFE servers must maintain, but it’s still something. In any event, if you’re using session state, you might want to use the session duration as your guideline for quantizing users to measure concurrency.

The script

Here’s the batch file with LogParser commands that I used to compute concurrent users by hour throughout the day. I know, I should have used PowerShell, but I only started with a couple lines and it just grew from there. Typical.

SET inputCSV=%1
if '%inputCSV%'=='' GOTO USAGE

REM outputCSV has no extension, it's added later
SET outputCSV=UserConcurrency

SET concurrencyPeriod=%2
if '%concurrencyPeriod%'=='' GOTO USAGE

SET concurrencyField=%3
if '%concurrencyField%'=='' SET concurrencyField=c-ip

REM Set a filter to match requests that should be excluded
REM %%Service%% matches our service accounts (like search), exclude them
REM ...if you don't want a filter, use SET filter=0 IS NULL
SET filter=(cs-username LIKE '%%Service%%')

echo.
echo Counting Concurrent Users
echo inputCSV : %inputCSV%
echo outputCSV : %outputCSV%.csv
echo concurrencyField : %concurrencyField%
echo concurrencyPeriod: %concurrencyPeriod% seconds

echo.
echo First stage, quantizing to %concurrencyPeriod% seconds...

logparser -i:CSV -o:CSV "SELECT DISTINCT %concurrencyField%, date, QUANTIZE(TO_TIMESTAMP(time,'hx:mx:sx'), %concurrencyPeriod%) AS Period, COUNT(*) as Requests INTO temp1-%outputCSV%.csv FROM %inputCSV% WHERE %concurrencyField% IS NOT NULL AND NOT %filter% Group By Date, Period, %concurrencyField%"

echo.
echo Second stage, grouping...

logparser -i:CSV -o:CSV "SELECT date, to_string(to_timestamp(Period,'hx:mx:sx'),'hh') as Hour, Period, count(%concurrencyField%) as UserCount, sum(Requests) as RequestCount INTO temp2-%outputCSV%.csv From temp1-%outputCSV%.csv Group by date, Period"

logparser -i:CSV -o:CSV "SELECT Hour, avg(UserCount) as Concurrent-Users(q%concurrencyPeriod%), sum(RequestCount) as Total-Requests(q%concurrencyPeriod%) INTO %outputCSV%-%concurrencyPeriod%.csv From temp2-%outputCSV%.csv GROUP BY Hour ORDER BY Hour"

GOTO DONE

:USAGE

echo.
echo # Usage:
echo #
echo # ConcurrentUsers inputCSV seconds [fieldname]
echo #
echo # inputCSV : csv file (or *.csv) of log entries with fields: date, time, c-ip or [fieldname], other
echo # seconds : concurrency quantization level. Typical values 300 and 600 seconds
echo # fieldname: field to evaluate for concurrency. Typical values c-ip (default) and cs-username
echo #
echo # Example : ConcurrentUsers BigO.csv 300
echo.

:DONE

Several things about this script.

The input is a CSV file, it must have fields named “date” and “time” at the very least, then another field named “c-ip” This field could have a different name, which you pass in as a parameter. This allows you to specify a different way to count users, such as cs-username, or maybe you want to count some other information quantized by time, maybe cs(User-Agent). I just default this to c-ip because I was working with a site that had anonymous access, and most of the cs-username values were null.

The file may have many other fields that come from the IIS logs.

The way I got my BigO.csv file was by following Mike Wise’s excellent whitepaper Analyzing SharePoint Usage: http://www.microsoft.com/download/en/details.aspx?id=4616. This will help you get started with LogParser in about 10 minutes (there’s that number again… an omen!). Some of his first steps show how to run LogParser on a whole set of raw IIS logs and collate them into a CSV file that has some extra info added, in order to do more analyses on that.

My script doesn’t require a CSV made in this way, this is just a good example of how to get one pretty easily from your logs. And of course, LogParser could just as easily be run directly against the logs, by using the option –i:IISW3C instead of –i:CSV. Mike’s paper talks about this too.

So, you can run this a few times and get results for different intervals that define your idea of concurrent use.

ConcurrentUsers BigO.csv 300

ConcurrentUsers BigO.csv 1200

ConcurrentUsers BigO.csv 3600

Running these in succession will output 3 files, UserConcurrency-300.csv, UserConcurrency-1200.csv and UserConcurrency-3600.csv, so you can easily compare for your analysis. It also creates some intermediate temp csv files that are useful to look into for sanity checking and understanding what the final output is telling you.

Oh, what’s that “SET filter=0 IS NULL” for? I just wanted an easy way to shut off the filter. Since the filter appears in the WHERE clause as AND NOT %filter%, I need something that evaluates to FALSE, meaning “don’t filter,” and as everyone knows, 0 is not null. Stupid geeks and their jokes.

Well, best of luck in this grey area. Let me know if any of this looks wrong please!!!

Control Search Indexing (Crawling) Within a Page with Noindex

Mark Arend — Mon, 07 Jun 2010 16:25:49 GMT

SharePoint provides the ability to control search indexing (a.k.a. “crawling”) within a page by using a div class named “noindex”. This means you can flag some content on a page that should not be indexed for search results, while other content on the page is still indexed. The typical scenario that requires this is common footer text. Suppose you edit your master page to contain some standard text, including your company name and a link to your privacy policy. Now, search results for your company or privacy policy will include every page in the site because this text is there. What you really want is just pages that contain unique content about those keywords.

SharePoint 2010’s iFilter excludes content inside of a <div class="noindex"> tag. Note that each div nested in that one is re-evaluated for the noindex class, so nested divs may not work as you expect. The product team is considering changing this so that all nested divs are excluded.

<div class="companyFooterStyle noindex">
Copyright 2010 © Company Name | <a href="/sitepages/PrivacyPolicy.aspx">Privacy Policy</a>
</div>

Now, when you search the portal for “Company Name” or “Privacy” you won’t get every page in the portal showing up in your results.

Note the ability to use other css classes along with noindex, to get visual styles as well. I haven’t tested the noindex class on tags other than div.

Technorati Tags: Sharepoint 2010,Noindex,Search

Using Session State in SharePoint 2010

Mark Arend — Thu, 27 May 2010 21:56:59 GMT

SharePoint 2010 uses two services related to session state that have similar names but are separate and have different purposes.

ASP.NET session state may be used on SharePoint 2010 pages. This service is automatically disabled in normal installations of SharePoint 2010; it may be enabled using the instructions below. Once enabled, it appears on the Service Applications page as “SharePoint Server ASP.NET Session State Service.”
The State Service service application is designed for and only available to Office internal components such as InfoPath Forms Services and Visio Services. This service is automatically enabled in typical installations of SharePoint 2010 and is not related to ASP.NET session state. It appears on the Service Applications page as “State Service.”

To enable ASP.NET session state:

Enter the following PowerShell command in the SharePoint 2010 Management Shell window:
```
Enable-SPSessionStateService –DefaultProvision
```
On each web application for which you want to use session state, edit the web.config file and set the enableSessionState property of the pages element as follows:
```
<pages enableSessionState="true"
```

Guidelines for using SharePoint Server ASP.NET Session State:

This session state may be used by custom code in web parts and other page-hosted controls.
Session state information is stored in a configurable SQL database; this implies the following:
1. Load-balancer affinity is not required.
2. Heavy use of session state must be included in database/capacity planning. By default, the session state database is created on the same database server hosting the SharePoint 2010 farm configuration database. This can be changed to place it on another server; this can be done at a later date if necessary. To move it to another SQL server, it should first be disabled, then enabled again with different configuration that points to the other server.
When enabled, Session state is available to the whole 2010 farm. Each web application can choose to enable it for use on ASP.NET pages (enableSessionState="true").
Session state is partitioned using the root URL. This means that different web applications will have different session state objects; a web part can only access the session state object available to it. If host header site collections are used, then different session objects will be used for each site collection (the browser will have multiple session ids in different cookies).

Additional information about enabling session state:

Syntax 1	Syntax 2
Enable-SPSessionStateService -DefaultProvision	Enable-SPSessionStateService -DatabaseName <String> [-DatabaseServer <String>] [-DatabaseCredentials <PSCredential>]

Additional parameters are not discussed here; use Get-Help Enable-SPSessionStateService –full for more information.

The Enable-SPSessionStateService cmdlet creates a session state database, installs the ASP.NET session state schema, and updates the Web.config files on the farm to turn on the session state service.

If the DefaultProvision form of the command is used, all default settings are used. These are:

DatabaseName = “SessionStateService_<GUID>”
DatabaseServer = the same database server as the SharePoint 2010 configuration database
DatabaseCredentials = Integrated Windows Authentication

If the DatabaseName form of the command is used, then these parameters may be set explicitly. If some of the parameters are not included, they default as above. To set specific credentials for accessing the session state database, note that this allows you to specify a SQL Authentication credential only, not a Windows credential. Then, use the Get-Credential command to create a PSCredential object from a username and password, then use that object as the argument of the DatabaseCredentials parameter.

After this service is enabled, “SharePoint Server ASP.NET Session State Service” will appear on the Service Applications management page.

Technorati Tags: SharePoint 2010,Session State,Development

Backup/Restore for SharePoint 2010

Mark Arend — Thu, 27 May 2010 21:46:39 GMT

This is an overview of major tools and techniques available to backup/restore your SharePoint 2010 farm

Using DPM to Backup/Restore SharePoint 2010

Microsoft System Center Data Protection Manager 2010 (DPM) is a product designed to manage ongoing, comprehensive backups of Microsoft enterprise products, including SharePoint 2010.

Protect at farm level – recover and track at granular level.
Automatic discovery & protection of new databases.
Seamlessly supports all SQL high-availability solutions.
Easy to configure and manage short-term and long-term backups
Optimized for large farms.
- Faster backups than SharePoint-only backup solutions.
- Requires significantly less storage than SQL-only backup solutions.
Search for SharePoint item through all recovery points of the farm.
Retains metadata/security settings for the recovered item.
Granular recovery without a recovery farm for SharePoint 2010.

More information on DPM: http://technet.microsoft.com/en-us/systemcenter/dm.

Backup/Restore for SharePoint using its own tools

SharePoint provides many of its own tools for backup and restore. This section discusses these options.

The Central Administration Backup and Restore page organizes SharePoint’s monitoring tools:

Central Administration provides tools for performing individual backups of the entire farm, individual site collections, sites, libraries or lists. A full farm backup can be restored from Central Administration, but tools are not provided here for scheduling recurring backups, or for restoring individual site collections, site, libraries or lists. These operations can be managed using PowerShell scripts:

Cmdlet	Synopsis
Backup-SPFarm (Restore-SPFarm)	Creates a backup of an individual database, Web application, or the entire farm. Can be full or differential. Use Backup-SPFarm -ShowTree to display a list of what would be backed up; this does not perform the backup.
Backup-SPSite (Restore-SPSite)	Performs a backup of a site collection.
Backup-SPConfigurationDatabase (Restore-SPFarm)	Performs a farm-level configuration-only backup. This is not a typical backup operation, see Note* below.

To schedule automated recurring backups, use these PowerShell cmdlets in scripts that are scheduled with the Windows Task Scheduler. This recommendation is from the TechNet article Backup and recovery overview (SharePoint Server 2010).

*Note that Central admin gives an option for backing up the config database, but you normally never restore the config database, even when restoring the farm. In DR scenarios, you create a new farm and restore the site collection DBs. An example use for a configuration database backup is to create a farm configuration template that can then be applied to create other new SharePoint farms that have identical physical topologies, by using Restore-SPFarm. This could be very useful in hosting scenarios with baseline virtual images.

SQL backup tools will process the data faster. DPM will process even faster and repeated backups require much less space.

So if you're planning daily backups of 500 GB, CA farm backup may be too slow. SQL backup will get it faster but require 2.5 TB to store 5 backups. DPM will be fastest and probably require less than 1 TB for 5 backups of 500 GB (depending on content churn).

Health Monitoring for SharePoint 2010

Mark Arend — Thu, 27 May 2010 21:28:31 GMT

This is an overview of major tools and techniques available to monitor the health of your SharePoint 2010 farm.

Using SCOM to monitor SharePoint 2010

Microsoft System Center Operations Manager 2007 (SCOM) is a product designed for enterprise health monitoring. It integrates closely with SharePoint 2010 and provides the most comprehensive and flexible solution for monitoring the health of SharePoint farms. With SCOM, the level of reporting and alerting is more granular and easily managed than SharePoint’s standard health monitoring.

SharePoint 2010 ships a management pack for System Center Ops Manager
Improved Knowledge Articles
More relevant events and monitors
Surfaces SharePoint Health Analyzer (SPHA) rules
Integrated with Unified Logging System (ULS)

The SCOM 2007 Management Pack for SharePoint 2010 contains more documentation for using these products together, in a Management Pack Guide for SharePoint Foundation 2010 and one for SharePoint Server 2010. Be sure to reference both guides for managing the server product. Download from: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c8a9d749-b7a8-412a-b2db-f3e464ed3fcf. These management pack guides are also included as a deliverable of this engagement, for convenience.

More information on SCOM: http://technet.microsoft.com/en-us/systemcenter/om.

Monitoring SharePoint with its own tools

With or without SCOM, SharePoint 2010 provides many of its own tools for health monitoring. This section discusses all of these options.

The Central Administration > Monitoring page organizes SharePoint’s monitoring tools:

Reports available from this page:

Health analysis problems and solutions
- Administrative configuration
- Performance issues
- Availability issues and
- Potential Security issues
- …results can be consumed by SCOM
Administrative reports
- Search administration (crawl rates, query latency, etc)
IRM Policy Usage reports
Health reports
- Slowest Pages
- Top Active Users
Web Analytics reports
- Traffic (Page Views, Visitors, Referrers)
- Search usage (query totals, trends)
- Inventory (Number of sites)

Custom reporting

Custom reports can be built and deployed to Central Admin Monitoring using Excel or SQL Reporting Services.

Custom health rules can also be created. Each custom health rule requires an assembly that will perform the desired health check. See How to: Create a Health Rule.

ULS logging service

SharePoint’s logging service (which uses the Windows Unified Logging Service or ULS) has been improved in several ways.

ULS Log Viewer
1. A SharePoint product team member discusses this tool on their blog: “In SharePoint 2010 we can use correlation id to trace a series of event inside SharePoint. For example, in this screenshot, my machine is trying to flush usage log. The different entries may be buried in the big ULS log file, but with correlation id this viewer can easily filter and find them all.”
2. Download ULS Log Viewer from: http://code.msdn.microsoft.com/ULSViewer
PowerShell cmdlets for managing the logs
1. Diagnostic Configuration
  1. Get-SPDiagnosticConfig retrieves Diagnostic Configuration values
  2. Set-SPDiagnosticConfig allows setting Diagnostic Configuration values
2. Trace Log and Event Log Throttling
  1. Get-SPLogLevel displays a list of diagnostic levels
  2. Set-SPLogLevel allows setting of trace/event level on categories
  3. Clear-SPLogLevel resets the trace/event levels to default
3. Log File control
  1. New-SPLogFile ends the current log and starts a new one
  2. Merge-SPLogFile combines trace log files from all farm servers into a single file
4. Trace Log Querying/Filtering
  1. Get-SPLogEvent reads and queries ULS trace logs
Correlation ID tracing
1. Every request has a unique correlation ID associated with all calls triggered by that request, even in SQL Profiler. Correlation ID appears in error message and can be used to find all associated actions immediately.
Manageability improvements
1. The ULS log file from each WFE server can now be directed to accumulate into a single SQL Server database. This makes it easier to find all relevant messages, and provides integration with SCOM monitoring. To enable this, click Configure usage and health data collection under Monitoring in Central Administration.
2. All events that appear in the Windows Application log also appear in the ULS logs, so no longer necessary to correlate from these two different sources.
3. Event flood protection. Heavily recurring events can be given a threshold so that they are only reported once every several minutes.

Developer Dashboard for Page Performance

This tool is useful for measuring the behavior and performance of individual pages. To analyze performance of a page, turn on SharePoint's "Developer Dashboard" that reports on details of page performance. Using stsadm commands for this:

stsadm –o setproperty –pn developer-dashboard –pv on

After this is executed, each page will show a report at the bottom that gives information about the performance of all the page calls and database queries.

Article: http://blogs.technet.com/speschka/archive/2009/10/28/using-the-developer-dashboard-in-sharepoint-2010.aspx

To cancel:

stsadm –o setproperty –pn developer-dashboard –pv off

To display an icon that allows you to toggle it on/off as you are working:

stsadm –o setproperty –pn developer-dashboard –pv ondemand

To set this parameter using PowerShell, use the following code snippet:

$DevDashboardSettings = [Microsoft.SharePoint.Administration.SPWebService]::ContentService.DeveloperDashboardSettings;
$DevDashboardSettings.DisplayLevel = ‘OnDemand’;
$DevDashboardSettings.RequiredPermissions =’EmptyMask’;
$DevDashboardSettings.TraceEnabled = $true;
$DevDashboardSettings.Update()

SPMonitoredScope

This class gives developers the ability to instrument their code so that it can be monitored using SharePoint tools, such as the Developer Dashboard. Example of use:

using (new SPMonitoredScope(
	“Some Scope Name”,
	TraceSeverity.Verbose, 1000,
	new SPRequestUsageCounter(3),
	new SPSqlQueryCounter())
	)
	{
		callExternalCode();
	}

Technorati Tags: SharePoint 2010,Health,Monitoring,SCOM

The Site Directory

Mark Arend — Wed, 26 May 2010 19:26:12 GMT

What happened to the Site Directory template in SharePoint 2010? This feature was deprecated by the product group, because new features in SharePoint 2010 were deemed a better way to automatically navigate and find sites and content, over the Site Directory model (which was admittedly a bit clunky, if you ask me).

So in SharePoint 2010, there is no longer an option to create a site with this template, and the portal site template does not include a Site Directory.

Now, if you upgrade a MOSS 2007 farm to SharePoint 2010, any Site Directory sites that you have will be upgraded and remain fully functional. So existing site directories are supported, but not the creation of new ones. This is why Central Administration still has an area to configure a Site Directory (under General Application Settings); so you may continue to manage any upgraded site directories correctly.

Bill Baer’s excellent blog gives some additional information on this topic, including the new features of 2010 that are envisioned to be better replacements, such as Tagging, Notes, Managed Metadata and Metadata-based Navigation.

Finally, for those who want the specific functionality of the Site Directory in SharePoint 2010, a CodePlex solution is available: http://spsitedirectory2010.codeplex.com/. This solution “follows the same basic principle of the Site Directory in SharePoint 2007 but adds some commonly requested features”

Technorati Tags: SharePoint 2010,Site Directory

SharePoint 2010 SDK

Mark Arend — Wed, 26 May 2010 17:52:48 GMT

The SDK (Software Development Kit) for SharePoint 2010 RTM’d slightly after the product itself and is now available. This download contains both SharePoint 2010 and SharePoint Foundation 2010 SDKs, which you may install separately. About 450 MB required to install both of them, but the download is much smaller.

Download the SDK now.

Tons of new “How-To” topics and code samples!

Code samples for SharePoint Foundation
- Implementation of IBackupRestore
Code samples for Business Connectivity Services
- AdventureWorks .NET Assembly
- AdventureWorks ASP.NET Web Service
- AdventureWorks WCF Service
- AdventureWorks 2008 .Net Assembly and Web Services
- AdventureWorks 2008 Declarative Outlook Solution Reusable Components
- AdventureWorks 2008 Declarative Outlook Solution Sample
- Authenticating and Consuming Data from NetFlix (Web 2.0 Application)
- BDC Model for XML Snippets
- Custom Web Part and Word Add-in Sample
- Sample Orders .Net Assembly and Web Services
- Sandboxed BCS Workflow Sample
Code samples for Enterprise Content Management
- Content Type-Based Metadata
- Creating a Routing Rule
- Custom Document ID Provider
- Managed Metadata Dependencies
- Pausable Custom Timer Jobs
- Publishing Cache Services
- Silverlight Controls and Silverlight Thumbnail Extraction Sample
- Retention Events
Code samples for User Profiles and Social Data
- Combine information from my public profile on Linkedin (or Facebook) with my User Profile
- Gold star or thumbs to a colleague
- Insert an event/document change event in feed
- Share a link with my colleagues
- Social data statistics
Code samples for Visio Services
- Annotations Sample
- Custom Error Messages Sample
- Mouse Interaction Sample

Technorati Tags: SharePoint 2010,Code Samples

Hands-On SharePoint 2010

Mark Arend — Wed, 26 May 2010 00:56:00 GMT

Some great resources are available for you to get hands-on tutorials and demo environments for SharePoint 2010 and Office 2010.

For IT Pros: http://MSSharePointITPro.com. This TechNet resource center is the starting place for understanding how to plan the physical architecture, infrastructure and configuration of SharePoint deployments.

Hands-On Lab: http://technet.microsoft.com/en-us/sharepoint/ff420396.aspx

For Developers: http://MSSharePointDeveloper.com. This MSDN resource center is the starting place for understanding how to plan development and implementation of custom SharePoint deployments.

Hands-On Lab: http://msdn.microsoft.com/en-us/sharepoint/ee513147.aspx

These labs let you get hands-on with an instance of SharePoint in the cloud, so no installation is necessary; you can get started right away.

SharePoint 2010 IW Demos: http://go.microsoft.com/?linkid=9728417. This is a Hyper-V virtual machine that has SharePoint 2010, Office 2010, Visual Studio 2010, SharePoint Designer 2010, SharePoint Workspace, and more. Plus, a ton of content is installed: sites, pages, users, documents, videos, workflows, libraries, etc., and the scripts that take you through how to use and demonstrate it all. This is an incredible resource. It’s also huge: 18 GB. But there is an optional link on the page to use Akami download manager to speed the download with pause/resume functionality. Requires Windows Server 2008 R2 with Hyper-V.

Corresponding documents:

SharePoint 2010 Overview Evaluation Guide http://go.microsoft.com/?linkid=9727161
SharePoint 2010 Walkthrough Guide http://go.microsoft.com/?linkid=9727427
Microsoft SharePoint Server 2010 Evaluation Guide For IT Professionals http://go.microsoft.com/?linkid=9727162
Microsoft SharePoint 2010 Professional Developer’s Evaluation Guide http://go.microsoft.com/?linkid=9727163

Technorati Tags: SharePoint 2010,Labs,Demos

SharePoint 2010 List and Library Size

Mark Arend — Wed, 26 May 2010 00:55:00 GMT

Great news about increased list capacities for SharePoint 2010:

Max recommended SharePoint 2010 list size is 30 million items or documents!

Plus, the maximum recommended number of items or documents in a view is now 5,000. Specific guidelines are given in SharePoint Server 2010 Capacity Management- Software Boundaries and Limits.

We still highly recommend nesting documents in a folder structure when you want to store millions of documents in the library. There are two new features to help you do this easily: the Content Organizer and Document IDs.

Content Organizer

This is a configuration that you can enable on a document library, to automatically organize documents into folder structures, even creating new folders when necessary. Steve Peschka describes it best in his detailed article on the Content Organizer:

“…it can be used for mundane housekeeping type activities such as making sure that no folder within a document library contains more than 5000 items, for example. When item number 5001 is added to the library, the Content Organizer can automatically create a new folder and put the document in that folder. All of this routing and folder creation can be handled automatically without further user intervention. After a document is uploaded and the required metadata applied, the upload form displays a Url to a document that has been routed so the user knows where to find it in the future.”

Document IDs

SharePoint now provides the ability to automatically assign a unique ID to documents as they are created or uploaded that can be used to locate them no matter where they are stored or moved in the farm. It works for single documents and the new Document Set. Blair Biggens discusses this in more detail.

Also very nice: it’s not a GUID but a bit more friendlier, shorter ID code!

SharePoint 2010 Content Database Size

Mark Arend — Wed, 26 May 2010 00:54:00 GMT

Great news about increased storage capacities for SharePoint 2010:

Max recommended SharePoint 2010 content database size is 200 GB!

This is double that of MOSS 2007, and for specific workloads—large, single-site repositories and archives with non-collaborative IO and usage patterns such as Records Centers—content databases up to 1 TB are supported. The specific guidance is given in SharePoint Server 2010 Capacity Management: Software Boundaries and Limits:

“We strongly recommended limiting the size of content databases to 200 GB to help ensure system performance.

“Content database sizes up to 1 terabyte are supported only for large, single-site repositories and archives with non-collaborative I/O and usage patterns, such as Records Centers. Larger database sizes are supported for these scenarios because their I/O patterns and typical data structure formats have been designed for, and tested at, larger scales.

“A site collection should not exceed 100 GB unless it is the only site collection in the database.”

A simple spreadsheet will help you plan your content databases and site quotas:

(open spreadsheet)

What enabled the size to increase? Two items that were (and remain) limiting factors have been improved. Understanding these is key to planning your SharePoint infrastructures.

List processing. The bulk of content in SharePoint is stored in lists and libraries. When many site collections are created in one content database, and collaborative activities grow across all the lists they contain, a very heavy load is placed on SQL Server. SharePoint 2010 contains improvements to list architecture and access mechanisms that address this.
Backup – The ability to backup and restore large databases in a reasonable time also limits how large you want databases to grow. SharePoint 2010’s farm backup and restore have performance improvements (mostly around multi-threading) that enable similar SLAs for larger database sizes.

Joel Oleson has a great article that includes more discussion about sizing, in SharePoint 2010 Capacity Planning.

SharePoint 2010

Mark Arend — Tue, 04 May 2010 18:36:52 GMT

After a long hiatus from blogging about MOSS 2007, I plan to begin again; now entries will focus on SharePoint 2010. This entry is more of a spacer than anything. That, and hopefully a kick in the pants to start posting new good info. Wish me luck!

Debugging web parts and other SharePoint custom code

Mark Arend — Tue, 09 Sep 2008 15:34:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

Here are some tips for debugging your SharePoint web parts and other custom code.

1. How do you debug assemblies that are in the GAC?

2. Is there an easy way of attaching the debugger to the right process?

Debugging assemblies that live in the GAC

When debugging an assembly that lives in the GAC, you have to copy the symbol file (.pdb) from your project to the GAC. The problem is, when you open a Windows file explorer to the GAC directory, C:\Windows\assembly, it doesn't behave like a folder that you can drag-and-drop into. You must copy using another method.

To view the GAC as typical folder structure using file explorer, enter %systemroot%\assembly\gac_msil in the Run... dialog box under the start menu. This folder opens the part of the GAC that contains managed code assemblies (from this location in file explorer, you can click the folder-up button to see other parts of the GAC).

From %systemroot%\assembly\gac_msil, locate the folder having the name of your assembly, and open that. Open the folder in there having the correct version number. Finally you’ll see the dll. To debug, drag the .pdb file from your bin\debug directory into that folder. You can also use an xcopy command to do this, now that you see the full path to the dll location. This path will change if you modify your assembly's version number. Note that whenever you upgrade your solution, the .pdb file is removed from that folder, so you have to copy it there again to debug again.

Of course you want to make sure to build in debug mode. Some projects are set up with dependencies on the \release\ directory, which I think is a bad idea, but it's a fact of life sometimes. If you don't want to change your build to Debug, you can quickly change your Release build to emit debug symbols: open the properties page for the project—make sure Configuration is Active (Release)—and on the Build tab, check “Define DEBUG constant” and uncheck “Optimize code” (you may also have to click the Advanced button and select "full" symbols). Now building in release mode will include debug symbols. Be sure to change these settings back once you’re done.

Attaching the debugger to the right process

A number of people have asked me about the "Attach Debugger" menu item they see me use. This is a community feature that's been around for more than a year, but it's good to refresh people's memory of this extremely useful tool.

A SharePoint product team blog entry discusses the Attach Debugger feature that you can install for SharePoint pages. It has a link to the CodePlex project SharePoint 2007 Features that you can install to enable this feature on the Site Actions menu of your sites:

Extremely helpful! And as a bonus, it seems faster than doing it from Visual Studio's Attach Debugger command.

I recommend you look at the SharePoint 2007 Features project, it's got lots of goodies besides this.

Technorati Tags: SharePoint Development

List of MOSS Timer Jobs

Mark Arend — Sat, 06 Sep 2008 16:26:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

MOSS has many default timer jobs; information about them can be useful to understand the workings of your MOSS farm, and potentially to troubleshoot problems or plan for related types of customizations.

The attached document provides a list of all MOSS timer jobs provided out-of-box, with some additional notes about them.

(Open Attachment)

Consolidated list of MOSS items

Mark Arend — Sat, 06 Sep 2008 16:16:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

Sometimes you just need a list of things provided by MOSS, in order to plan what kind of things you will make available to users within your architecture.

The attached document lists all out-of-box Lists, Web Parts, Site Templates and Site Actions provided out-of-box by MOSS, along with their descriptions and availability.

(Open Attachment)

MOSS Usage Reports explained

Mark Arend — Sat, 06 Sep 2008 15:55:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

The site usage reports that MOSS provides are fairly simple, but can be useful if you haven't got any other tools for tracking site usage. One problem with them, however, is that the information they present is not really explained anywhere. Columns are often named "Value" and sometimes it's not clear exactly what the value represents. Another example is what does "Requests" actually measure? The answer is that Requests always measures Page Views, not all HTTP requests for individual items like images, style sheets, etc.

The attached document gives clear, detailed explanations of all out-of-box report elements.

(Open Attachment)

Colleagues, Social Distance & Relevance in People Search; Social Networking tools

Mark Arend — Thu, 01 May 2008 16:16:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

MOSS provides some very powerful features to enable the new buzzword "social networking." Discussions about these can be found pretty easily, and I give an overview of some tools near the end of this posting. But specifics about how some of the details work has been devilishly hard to find... until now. Through discussions with various people in-the-know, I've been able to assemble answers to common questions around People Search and MySite web parts.

Social Distance

By default, the results on the People Search page are ordered by Social Distance. What the heck is that? It's an ordering of results based on colleague relationships. Here's how it's computed:

Search results are always returned to the search results page sorted by relevance.
When viewing results by social distance, additional processing on the search results page is used to group the results:
- The first 3 pages of search results are grouped by colleague-ness: first your colleagues appear, then colleagues of your colleagues, then everyone else.
- Within each group, the ordering is still by relevance.
- When paging through results, another 3 pages of results will be grouped once you reach page 4, then page 7, etc.

By default, 10 results appear per page, so this groups people in batches of 30. If you customized the People Search results page to show 25 results per page, this would group people in batches of 75.

There are potentially some quirks with this algorithm that was designed to give the best value in a lightweight manner. Your colleagues will appear at the top of the list on page 1, but if you have lots of colleagues, more of them might appear on page 4. This is because the colleague grouping takes place on the client, starting with a relevance ranking. So if some of your colleagues are relevance-ranked after 3 pages worth of results, these will be grouped once you get to the next set of 3 pages (page 4, page 7, etc). Based on relevance, these people should be less relevant to your query.

The People Search results page allows you to toggle the results ordering to only Relevance, ignoring the Social Distance grouping.

Relevance

Relevance ranking in people search is the same as that used in regular search across lists and documents. I'm told that the ranking algorithm MOSS uses in all cases is BM25F.

Conceptually, a user's profile is treated as a document about that person with their name as the title. At a basic level, the more search query terms that appear in the person's profile, the more relevant the result. Matches against a few special fields (e.g. name, alias) are returned in the high confidence webpart.

Note that the names of discussion lists that a user belongs to are included in that user's profile information. This means that a people search for "rock climbers" should return all of the people who are members of a distribution list named "Rock Climbers at Contoso" as well as anyone who has the phrase "rock climbers" in their profile information, such as in their "About Me" field.

Colleagues

Here's where the real value of social distance and other meaningful relationships is computed.

A list of your colleagues is stored in the profile database. During a profile import, this list is initially populated with a set of "immediate colleagues" that are computed from your profile properties: manager, peers and direct reports. This way, when you first look at your Colleagues web part, it's not blank.

Then, from an editing page on your Profile, you can edit your colleagues list to add & remove specific people, and to group them and set their visibility to other people. Also from here, you can ask SharePoint to suggest other colleagues, and these suggestions are collected from several places.

My Microsoft badge photo, 1999!

From your My Site page (or your Profile page, which is available even if My Sites are turned off), click Colleagues to edit your colleague list.

Where do colleagues come from?

Immediate colleagues
...automatic: during profile import; comprised of:
- Your manager
- Your peers (others who report to your manager)
- Your direct reports
Colleagues added by you
...manual: via editing page on your Profile
Suggested colleagues
...lookup: manually guided via editing page on your Profile; comprised of:
- Sent items in Outlook. Periodically (every 5 days or so, depending on usage) Outlook analyzes Sent Items for common recipients, weighted by frequency of contacts and other factors. This is how Outlook is able to suggest likely recipients when you begin typing their name in the To: line, and in my experience, it seems to work pretty well. When you click Suggested colleagues, it collects these names from Outlook via an ActiveX control, and adds them to the list of suggestions.
- Office Communicator contacts are examined; that is, those contacts that you've added to your Office Communicator client from the company directory. These are also collected from the Communicator client via an ActiveX control.
- Site Memberships are all sites in which you are explicitly included in the "~Members" group for the site (where ~ is your site name). Other users who are also explicitly included in the ~Members group on those sites are suggested as your colleagues. This information is collected by a WSS profile synchronization timer job.
- DG (Distribution Group) Memberships are examined for suggestions. Distribution groups stored in AD are collected during the AD import.

Colleagues from these sources are collected and the top 20 or so are displayed as suggestions when you click "Add Colleague" on your colleague editing page. It's not fruitful to pursue a more detailed understanding of this calculation, because the recommendation for improving the "quality" of colleagues found is the same: people associated with items you access more often are more likely to be suggested as your colleagues. If you don't like the suggestions, you can remove individuals from the suggestion list one by one or all at once from this page, then get a new list of suggestions.

Privacy

Whoa! Is SharePoint looking at my email? I don't want that, how do I prevent it?

First, SharePoint's not looking at your emails. It gets information about who you sent emails to by asking Outlook for that information. Second, it will never do this without asking your permission. You can always say no:

If you say Yes, then SharePoint will assemble a list of suggested colleagues for you, and you can accept or reject these suggestions. But even if you say No, you can still edit your colleagues to search for and add new people, and to set the privacy of who should see that they are your colleague (Everyone, My Colleagues, My Workgroup, My Manager and Only Me):

Social Networking tools

Why did we go through all this trouble to try and figure out who your colleagues are? Social Networking, baby! We spent a good bit of time on the inputs and the outputs of this to make it a really useful function, instead of just paying lip service to it.

The inputs are the information about who your colleagues really are. A simple option would have been to just provide a place for you to register your colleagues, and not go through all this supposedly intelligent guesswork. But that would have made it almost useless, because most people are just not going to take the time to populate this list, and even fewer will maintain it as their colleagues change over time, which is natural.

SharePoint uses the information already embedded in your company's infrastructure and the changing data you deal with on a daily basis to help you connect with the right people.

Sounds like market-speak, but that's really the goal and I think it does an amazing job.

The outputs are the tools that SharePoint can provide based on this colleague information. Consider some of these details:

People Search ranking by Social Distance. By default, the top people returned when you search for a name will be the people you most likely want to find: people you've communicated with recently, perhaps as a participant on an email thread or a collaborator on a site, and now you want to find more information about them.
In Common with You web part on My Profile page. When you view someone else's profile (for example, by clicking their People Search result), this web part shows you a list of people that you both may have in common. This is useful to quickly understand business relationships that you may need to know about. Categories shown are
- Manager we both report under
- Colleagues we both know
- Memberships we both share
Colleague Tracker web part on MySite. This shows you recent changes to user properties of your colleagues, effectively a newsletter of what's new with people you work with. Information you can track includes
- Anniversaries
- Profile property changes
- Membership changes
- New documents this person has posted
- Out of Office status
- Blog postings from this person
Colleagues web part on My Profile page. This displays your current list of colleagues, grouped into categories that you may have applied.
Memberships web part on MySite. This shows Distribution Lists and SharePoint Sites of which you are an explicit member. For SharePoint Sites, it is rather specific: you must appear as an individual member of the ~Members group on the site (where ~ is the site name). If you have access to the site through a different configuration, such as being a member of the ~Owners group, or being a member of an Active Directory Security Group that is a member of the ~Members group, this is not counted. Otherwise, everyone would very likely see all the portal sites displayed as their memberships, and that would not be useful.
Organization Hierarchy web part on My Profile page. This shows your manager, your peers (others who report to your manager) and your direct reports, if you have any.

These are some of the tools that MOSS provides for Social Networking; I've highlighted the ones that use profile data and the powerful "colleague" pattern to show meaningful relationships among people. For a better introduction to the full set, see the product team's Enabling and Managing Social Networks for Business use with MOSS.

Technorati Tags: SharePoint,Social Networking,MySites,People Search,Social Distance

Scaling My Sites; How Many My Sites per Content DB?

Mark Arend — Tue, 15 Apr 2008 00:43:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

I've worked with several companies who have between 100,000 and 200,000 employees. Naturally, they want to architect a solution for MySites that can accommodate everyone in the company. TechNet has information in the Plan for software boundaries article, but a few numbers are ambiguous or conflicting. In this post I'll try to clear some of that up.

The limits I'll respect from this article are:

100 GB maximum size for Content Database (DB)
50,000 site collections per Content DB

Scale-out

The primary scale-out strategy is to use multiple content databases in the MySite web application. To meet the guideline of 50,000 site collections per DB, you should plan a minimum of 2 content databases to handle 100,000 MySites.

Next, consider Throughput vs. Site Collections discussed in the TechNet article. The most striking visual impact of this graph (reproduced below) is that performance appears to drop off dramatically as the number of site collections per content database grows to 50,000. It looks a bit alarming. But it's important to examine the numbers.

Let's look at some points on the graph. Note this is a sample from testing; your specific numbers may vary, but we'll assume inflection points are in roughly the same places.

At 10,000 site collections per content database, the number of requests per second (rps) is still above 100.
From about 14,000 - 16,000 site collections, throughput decreases very rapidly.
After 16,000 site collections, throughput "bottoms out" near 50 rps and stays pretty much the same through 50,000 site collections.

As a general approach to optimizing throughput for any arbitrary scenario, we might say that limiting content databases to 10,000 site collections is a good rule of thumb, because adding 50% more databases at this point cuts throughput in half. But you should think through anticipated usage patterns for the particular web application and site collections.

MySites have a common usage pattern: these site collections tend to have relatively small quotas, and don't get a lot of concurrent traffic. A relatively small percentage of MySites are used heavily; these tend to be people who enjoy publishing material or experimenting with information techniques. In many situations, 50 rps will be sufficient for MySite traffic. This means we're free to set the limit of site collections per content database on the MySite host at any value up to the recommended limit of 50,000.

The next limiting factor is the overall size of the content database: 100 GB is recommended as the maximum. This, plus the desired MySite storage quota, will let us compute the optimum number of databases.

Individual MySite quota (MB)	2	5	10	20	25
Number of MySites per 100 GB content DB	51,200	20,480	10,240	5,120	4,096
# content DBs per 100,000 users	2	5	10	20	25

Cool Rule: Assuming 100GB as the maximum size of a content database, the number of databases you should plan for—per 100,000 users—is equal to the quota of each site collection in MB!

Other combinations of maximum content DB size or number of users will not line up this way. To compute the number of content databases, the following formulas may be used.

# site collections per DB = (Max size of DB in GB * 1024) / (MySite quota in MB)
# content DBs = (Number of MySites) / (# site collections per DB)

For the second formula, you must always round fractions up to the next whole number (use the ceiling function in Excel).

Scale-up

To further ensure the overall performance of SharePoint when using multiple content databases, configure SQL Server so that the databases are hosted on different physical drives (spindles). This is usually possible with SAN storage solutions also. My instinct is that you'd want to balance this with the scale-out strategies and usage patterns. If you had 200,000 users with MySites and a large portion really began to use MySites heavily, then you might want to use 20 content databases distributed across 5 or 10 physical drives.

Next, note that MySites usually cause significantly more transactions in the User Profile database than collaboration or publishing sites, because more web parts that present and manipulate profile information are available on MySites. As you grow the number of MySites, the CPU, memory, storage and IOPS available for your User Profile DB becomes more important. Therefore you want to consider putting this database on a separate physical drive also. For very large user populations and/or heavy anticipated use of MySites, you could even consider using a separate instance of SQL Server for the Profile DB.

The whitepaper SharePoint database performance recommendations gives specific recommendations for increasing performance via the database.

What about TechNet’s limit of 150,000 site collections per web app? Well, as of this writing, the article shows two different numbers for the same limit (the other is 50,000)... clearly, there is still some ambiguity over this. Recent anecdotal evidence hints that 200,000 site collections can be hosted on one web application and still perform well. If I find more details about this I'll post another entry.

Areas a performance hit might be seen due to larger numbers are those that enumerate site collections in a web app. There are pages in Central Administration (CA) that do this, and it can be mitigated by using the "find" function in the CA page or by using stsadm commands to perform some functions such as MySite deletion.

If you want to plan for growth toward heavy MySite usage, you may want to consider using more than one web application to host them. It's possible to configure multiple web applications to host MySites. When doing this, audiences can be defined to map users to MySite web apps. A very common approach is to use geography as the audience, especially when a company has datacenters around the world that service users in their part of the world. This has the added benefit of hosting a user's MySite from a datacenter nearer to them. Refer to TechNet article Manage My Site host locations for much more detailed information on enabling this scenario.

However, this may not be as balanced an approach for a domestic as it is for global companies. The ideal arrangement is to find a natural organization of people that splits the whole population into just a few categories of reasonably equal quantities, and is reflected in user profile data. Such a perfect arrangement may not exist; review your user profile database to look for natural categories that can uniquely classify everyone.

Technorati Tags: MySites,SharePoint Capacity

Modifying My Site Owner Security

Mark Arend — Thu, 10 Apr 2008 07:57:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

When someone creates a MySite, they are automatically given full control of that site. More accurately: a MySite is a site collection; the creator becomes the first & only Site Collection Administrator; the creator is made a member of the Owners group.

Some people feel that this is too much control, and they'd like to limit it in some way. My first piece of advice in this situation is: don't. MySites are built with dependencies on the permissions and identity of the owner, the person who created the site. Changing this can introduce undesirable or confusing behavior. Then, if requirements later change, after hundreds or thousands of people have created their MySite, more code must be written to effect those changes. But if you insist on going down this path, I found a way to meet this seemingly simple request. The answer is most certainly not simple.

Basically, you have to follow Steve Peschka’s approach described in the blog entry customizing My Sites as the framework for doing this. Then, add code to that scheme that knows how to change the security. There are a few added challenges:

You must do two things: 1) change the creator’s group membership and 2) remove them as a Site Collection Administrator.
To give the creator of the MySite specific privileges, you typically want to create a custom privilege level and add the user into that group. The benefit is that you can tweak this to allow/prevent actions at a fairly granular level, such as being able to block the ability to add subsites.
The creator of the MySite is the only Site Collection Administrator, so you can’t remove this person from this role until you first put someone else in there. For best governance, this should be a user account that’s not really associated with an individual, but shared among trusted admins. Unfortunately it must be a user account, not a group.
There seems to be a timer job that fixes up the title of the MySite to match the name of the site administrator… so you need some code that resets this back every time the user opens the site.

Why can’t we just change the MySite site definition (template)? Two reasons:

Modifying the MySite site definition is “unsupported” by Microsoft. If you did it, then wanted help with problems related to it, the support engineer would make a best effort, but could not guarantee a fix. Steve’s article talks more about this.
Moot point! User security is not stored in the site definition, so even if it were supported to modify it, there would be nothing to do.

Below are code samples that address the challenges listed above. Your job would be to integrate them with Steve Peschka’s code. Sorry I can’t give a more complete solution. This is one task that is very difficult to accomplish in SharePoint, and the methods described are the only way we know of doing it in a supported way.

(BTW, I use lots of try...catch blocks here to get the most granular logging. You can implement Utility.LogMessage any way you like; I use MSDN's Trace Log Example to write to the ULS logs, and CodePlex's Log Viewer Feature to read them.)

public void RestrictMySiteOwner()
{
    SPUserInfo originalOwnerInfo = new SPUserInfo();
    SPUserInfo newOwnerInfo = new SPUserInfo();

    // Store the original owner in site properties so we can restore it later if necessary
    try
    {
        StoreValueInSiteProperty(_curWeb, "OriginalOwner", MakeUserInfoString(originalOwnerInfo));
        Utility.LogMessage("Stored original administrator in site property.");
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't store original administrator in site property.  " + ex.Message);
    }

    // Switch out the primary site administrator with a pre-defined one.
    try
    {
        newOwnerInfo.LoginName = "DOMAIN\MySitesAdmin";
        newOwnerInfo.Email = "mysitesadmin@customer.com";
        newOwnerInfo.Name = "MySites domain manager";
        newOwnerInfo.Notes = "Owner of all MySites";
        originalOwnerInfo = ReplacePrimarySiteAdministrator(_curWeb, newOwnerInfo);
        Utility.LogMessage("Switched primary administrator.");
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't replace primary administrator.  " + ex.Message);
    }

    // Add the My Site creator to a group with limited permissions to control what is permissible on the site
    try
    {
        ChangeCurrentUserPermission(_curWeb, "Restricted Owner");
        Utility.LogMessage("Changed current user permissions.");
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't change current user permissions.  " + ex.Message);
    }

    // Reset the name of the site back to original owner (was changed by switching primary site admin)
    try
    {
        SetSiteTitle(_curWeb, originalOwnerInfo.Name);
        Utility.LogMessage("Reset site title.");
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't reset site title.  " + ex.Message);
    }
}


protected SPUserInfo ReplacePrimarySiteAdministrator(SPWeb site, SPUserInfo newAdminInfo)
{
    SPUser originalOwner = null;
    SPUserInfo originalOwnerInfo = new SPUserInfo();
 
    SPSecurity.RunWithElevatedPrivileges(delegate()
    {
        // Get the parent site collection
        SPSite sitecollection = site.Site;

        originalOwner = sitecollection.Owner;
        originalOwnerInfo = Utility.GetUserInfo(originalOwner);

        if (newAdminInfo.LoginName != originalOwnerInfo.LoginName)
        {
            try
            {
                // Add new admin to Full Control group
                SPRoleDefinition admins = site.RoleDefinitions["Full Control"];
                if (admins != null)
                {
                    SPRoleAssignment roleAssignment = 
                        new SPRoleAssignment(newAdminInfo.LoginName, newAdminInfo.Email, 
                        newAdminInfo.Name, newAdminInfo.Notes);
                   SPRoleDefinitionBindingCollection roleDefBindings = 
                        roleAssignment.RoleDefinitionBindings;
                    roleDefBindings.Add(admins);
                    site.RoleAssignments.Add(roleAssignment);
                    //site.Update(); // Don't need to do this
                }

                // Reset owner of site collection to new admin
                sitecollection.Owner = site.Users[newAdminInfo.LoginName];
                sitecollection.Owner.Update();

                //change the userprofile guid of the site
                sitecollection.Properties["urn:schemas-microsoft-com:sharepoint:portal:profile:userprofile_guid"] = toProfile.ID.ToString().Replace("{", string.Empty).Replace("}", string.Empty); 
                sitecollection.Properties.Update(); 

            }
            catch (SPException ex)
            {
                // We may get here if the user running this thread is the same as the 
                // aministrator we're trying to remove.
                Utility.LogMessage("Error: can't replace primary administrator.  " + ex.Message);
            }
        }
    });

    return originalOwnerInfo;
}


protected void ChangeCurrentUserPermission(SPWeb site, string roleDefinitionName)
{
    SPUser curUser = site.CurrentUser;
    string curUserName = curUser.LoginName;

    SPRoleDefinition roleDefFull = null;         // Full control privilege level
    SPRoleDefinition roleDefContributor = null;  // Contributor privilege level
    SPRoleDefinition roleDefLimitedOwner = null; // Our new privilege level
            
    // Create new role definition
    try
    {
        // Get existing role definitions
        roleDefFull = site.RoleDefinitions["Full Control"];
        roleDefContributor = site.RoleDefinitions["Contribute"];

        // Create custom role definition
        roleDefLimitedOwner = new SPRoleDefinition();
        roleDefLimitedOwner.Name = roleDefinitionName;
        roleDefLimitedOwner.Description = "Permission settings for the owner of this MySite.";
        site.FirstUniqueRoleDefinitionWeb.RoleDefinitions.Add(roleDefLimitedOwner);
        site.FirstUniqueRoleDefinitionWeb.Update();

        // Get the new role definition again... makes sure the Update took all the way or something.
        roleDefLimitedOwner = site.FirstUniqueRoleDefinitionWeb.RoleDefinitions[roleDefinitionName];

        // Set permissions of new role... start from Contributor as base.
        roleDefLimitedOwner.BasePermissions = roleDefContributor.BasePermissions;
        // To ADD a permission, use Bitwise-Or assignment:       permissions |= permission
        // To REMOVE, use Bitwise-And assignment to Complement:  permissions &= ~permission
        roleDefLimitedOwner.BasePermissions |= SPBasePermissions.CancelCheckout;
        roleDefLimitedOwner.BasePermissions &= ~SPBasePermissions.AddDelPrivateWebParts;
        roleDefLimitedOwner.BasePermissions &= ~SPBasePermissions.UpdatePersonalWebParts;
        roleDefLimitedOwner.Update();
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't create new permission group \"" 
            + roleDefinitionName + "\". " + ex.Message);
    }
            
    // Add owner to new role definition
    try
    {
        SPRoleAssignment assignment = new SPRoleAssignment(curUser);
        if (roleDefLimitedOwner != null)
        {
            assignment.RoleDefinitionBindings.Add(roleDefLimitedOwner);
        }
        else
        {
            //Workaround: for now we just make owner a contributor
            assignment.RoleDefinitionBindings.Add(roleDefContributor);
        }
        site.RoleAssignments.Add(assignment);
        //site.Update(); // Don't need to do this
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't add user \"" + curUserName 
            + "\" to new permission group \"" + roleDefinitionName + "\".  " + ex.Message);
    }

    // Remove owner from Full Control
    try
    {
        SPRoleAssignment userRoleAssignment = UserRoleAssignment(site, curUser, roleDefFull);
        if (userRoleAssignment != null)
        {
            userRoleAssignment.RoleDefinitionBindings.Remove(roleDefFull);
            userRoleAssignment.Update();
        }
        //site.Update(); // Don't need to do this
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't remove user \"" + curUserName 
            + "\" from default permission groups.  " + ex.Message);
    }
}
 
 
private SPRoleAssignment UserRoleAssignment(SPWeb site, SPUser curUser, SPRoleDefinition curRoleDef)
{
    foreach (SPRoleAssignment roleAssignment in site.RoleAssignments)
    {
        if (roleAssignment.Member.ID == curUser.ID)
        {
            foreach (SPRoleDefinition roleDefinition in roleAssignment.RoleDefinitionBindings)
            {
                if (roleDefinition.Id == curRoleDef.Id)
                {
                    return roleAssignment;
                }
            }
        }
    }
    return null;
}


protected void StoreValueInSiteProperty(SPWeb site, string propertyName, string propertyValue)
{
    if (site.Properties.ContainsKey(propertyName))
        site.Properties[propertyName] = propertyValue;
    else
        site.Properties.Add(propertyName, propertyValue);
    site.Properties.Update();
}


protected void SetSiteTitle(SPWeb site, string title)
{
    try
    {
        site.Title = title;
        //site.Update(); // Don't need to do this
    }
    catch (Exception ex)
    {
        Utility.LogMessage("Error: can't set site title to \"" + title + "\".  " + ex.Message);
    }
}

Technorati Tags: MySite,SharePoint Security,SharePoint Development

New Design and Build Guide for MOSS 2007

Mark Arend — Wed, 12 Mar 2008 21:36:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

This is a topic near and dear to my heart. I've spent many hours helping customers plan and execute a wide range of customizations for MOSS 2007.

Now, the ECM Team (Enterprise Content Management) has published an exhaustive paper on these topics. At least, I was exhausted reading it! No, really, it's a great guide and covers so many areas that normally don't even get touched on.

Download: http://go.microsoft.com/fwlink/?LinkId=110087.

Announcement and request for feedback:
http://blogs.msdn.com/ecm/archive/2008/03/11/announcing-design-and-build-sites-for-office-sharepoint-server-2007.aspx.

Topics

Customization approaches (Basic, Intermediate, Advanced - Author centric, Advanced - Developer centric)
Site Elements (Authored elements, Developed elements)
Environments (Dev, Config mgt, Integration, Authoring, Pilot, Production)
Tools & Processes
Development team roles (Architect, Developer, Designer, Author, Tester, Admin)
Customization Scenarios (Corporate extranet, Enterprise internet)
Upgrading from SPS 2003 to MOSS 2007 (addressing customizations)
Upgrading from CMS 2002 to MOSS 2007 (pointers to resources)
Deployment (Authored, Developed)
Solution packages
Roadmaps (Sites & Subsites, Page design, Document mgt, InfoPath, Workflows, BI, Records mgt)

This is a tremendously valuable resource, go get it!

Technorati Tags: Development,MOSS 2007

Test With Correct Privileges

Mark Arend — Fri, 07 Mar 2008 00:12:00 GMT

When developing web parts and other components with code, it's important to remember that operations performed by code may require certain privileges. During development, the code runs in the security context of the developer, who is usually a server administrator. This means all operations are permitted, so access problems don't arise. During testing and production, however, the security context of the code may be different because it's being invoked by testers and end-users.

Therefore, all components should be tested with several end-user accounts: one would be the developer’s account which is usually a server admin; that’s how most people do their unit testing. But tests should also be made with an account of minimal privilege, say Visitor. Some components might be doing things with the object model that will only work when an administrator invokes it. If this turns out to be the case, a technique called “elevation of privilege” can be used to overcome it. See MSDN article Running Commands with Elevated Privileges in Windows SharePoint Services 3.0 for an explanation of doing this.

What code needs to be tested using different privileges?

1. Code that's invoked as the result of an end-user action. An example is a custom web part that is displayed on a page. The end-user opens the page and this is what invokes the web part code under the user's security context. A counter-example is timer job code that is invoked by SharePoint.

2. Code that uses the SharePoint API. This is typically where security errors arise. If a component does not use the SharePoint API, then chances are it will not need to elevate its privilege. To be entirely sure, it should be tested at minimum privilege, but this criterion can be used to prioritize testing of other components higher.

How can a developer test with different accounts?

The simplest approach is to create three accounts "Test Visitor," "Test Member," and "Test Owner" in Active Directory and then to add these accounts to matching groups on the test sites.

First, the developer must decide in what security contexts the component is supposed to work. For example, most end-user web parts should work for users signed on as Test Visitor. On the other hand, an stsadm extension is only invoked by a server admin and should not be tested with a minimum privilege account.

"Sign on as different user" can be used to choose one of the test accounts as appropriate; however, in some cases, this approach does not seem to “stick” based on browser state or browsing behavior (e.g. browsing to My Site when browser is configured for automatic logon). The best way to ensure that the desired account is used is to configure the browser to always prompt for credentials, then to open a new browser window to begin testing. Then, re-run all unit tests that exercise the component directly (not, say, the deployment of the component).

Technorati Tags: Minimum privilege,Web part security,Code Access Security

Virtual PC and Virtual Server

Mark Arend — Wed, 05 Mar 2008 04:58:00 GMT

Wait.. A posting not about SharePoint? How can this be? Well, it's somewhat related to SharePoint, since so many people use Virtual PC to host a development SharePoint environment. Since SharePoint requires Windows Server 2003, that's the best choice for us laptop users. Anyway, this is just a quick comparison between some points of these two products.

Feature	VPC	Virtual Server
Shared Folders?	Yes	No
Copy-n-paste?	Yes	No
Drag-n-drop?	Yes	No
Map a virtual NIC to a wireless physical connection?	Yes	No
Compact disk image to a separate location?	Yes	No
Management tools for multiple guest OS?	No	Yes
Performance	Great*	Great*+1
Runs as Windows Service (can auto-restart after host restart)	No	Yes

*Performance really can be very good, on two conditions: enough physical RAM on your host (2GB will do, but 4GB is much better), and putting the virtual hard drive on a physical drive other than the one holding the host OS. On a laptop, a USB 2.0 external drive works very well. Be sure to "safely remove..." the USB drive before unplugging it.

Bottom line: if your host is a laptop or desktop, use VPC. If your host is a server in the datacenter, use Virtual Server.

Sometimes people ask me about VMWare's products. Well, I don't run them so I can't say, but Andrew Connell has a good blog posting examining some of the differences, especially related to how we use them as SharePoint developers, consultants and trainers.

Technorati Tags: Virtual PC,Virtual Server

Using Reflector to See SharePoint's "Source" Code

Mark Arend — Fri, 29 Feb 2008 00:49:00 GMT

Important Note: Customers are advised to review the end user license agreement (EULA) for the relevant Microsoft products before using this tool. This blog entry does not authorize or endorse any use of Microsoft products that is not addressed specifically by the product's EULA.

Do you sometimes use a "reflector" program to examine SharePoint source code? This can be necessary to:

Understand out how SharePoint is doing something, to find what elements can be customized.
Understand what members to override in classes you create that inherit from SharePoint classes.

The de facto standard program for doing this is Lutz Roeder's Reflector. The MSDN article on ten must-have tools has a great description of how helpful it can be and how to use it.

One thing the article doesn't tell you is where to find SharePoint's assemblies; fortunately it's not that hard. Click "Open" in Reflector, then with a little bit of browsing, most people easily find the dlls in the ISAPI directory of the "12-hive" (C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\ISAPI).

Open one or more of these dlls, then browse the classes to see how SharePoint's put together… the Disassemble tool is exceedingly useful! Note: the code you see is reconstructed from MSIL, into the language of your choice (C#, VB, Delphi, C++ or Chrome)… the product teams didn't really use all those goto's!

However, some SharePoint namespaces seem to be missing or incomplete when viewed in Reflector. This had me baffled for quite some time, until I found the following.

Some SharePoint assemblies are stored in the _app_bin directory of the web application instead of the ISAPI directory in the 12-hive. Presumably, the reason for distributing certain code into the web application's root is related to something like stability, security or scalability. But the important point is if you're using Reflector to look at SharePoint code, you may need to look here as well.

This is especially true for a namespace like Microsoft.SharePoint.ApplicationPages. Two assemblies contribute members to this namespace: 12\ISAPI\Microsoft.SharePoint.dll and Inetpub\...\_app_bin\Microsoft.SharePoint.ApplicationPages.dll. If you just reflect the one in ISAPI, you'll see very few members; the ones you typically want to look at are in _app_bin!

SharePoint DLL locations

Here's a list of all the SharePoint assemblies and where they appear.

12\ISAPI

Microsoft.Office.Excel.Server.Udf.dll
Microsoft.Office.Excel.Server.WebServices.dll
Microsoft.Office.Policy.dll
Microsoft.Office.Server.dll
Microsoft.Office.Server.Search.dll
microsoft.office.workflow.tasks.dll

Microsoft.SharePoint.dll
microsoft.sharepoint.portal.dll
Microsoft.SharePoint.Portal.SingleSignon.dll
Microsoft.SharePoint.Portal.SingleSignon.Security.dll
Microsoft.SharePoint.Publishing.dll
Microsoft.SharePoint.Search.dll
Microsoft.SharePoint.Security.dll
microsoft.sharepoint.WorkflowActions.dll

Not .NET assemblies (can't reflect):
OWSSVR.DLL
SHTML.DLL
SSOCLI.DLL

Inetpub\wwwroot\wss\VirtualDirectories\<web app>\_app_bin

Microsoft.Office.DocumentManagement.Pages.dll
Microsoft.Office.officialfileSoap.dll
Microsoft.Office.Policy.Pages.dll
Microsoft.Office.SlideLibrarySoap.dll
Microsoft.Office.Workflow.Pages.dll
Microsoft.Office.WorkflowSoap.dll

Microsoft.SharePoint.ApplicationPages.dll

stssoap.dll

Obfuscation tips

Sometimes, not too often, when Reflector is disassembling source code for you, it will show "This item is obfuscated and can not be translated" instead of code. I think the product team did this for some of the more "sensitive" areas of the code (such as LogInAsAnotherUser) and also on a bit of a random basis, perhaps to prevent fully reverse engineering the product. In any case, by changing the translation language to IL in Reflector, you might get enough information from the MSIL code to answer the question that drove you there in the first place. For example, sometimes I just need to know what modules are being called, or what strings are being used to find other resources in the 12-hive; these are still visible in MSIL.

Important Note: Customers are advised to review the end user license agreement (EULA) for the relevant Microsoft products before using this tool.

Technorati Tags: SharePoint,MOSS+2007,Development

My Site Pages and Architecture

Mark Arend — Sat, 23 Feb 2008 04:03:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

MySites are very interesting on many levels. When you start to think about how to architect, deploy or customize them, you may encounter some initial confusion and a lack of detailed information. For instance, did you know that your MySite's Home page and the My Profile page are in completely different site collections? What's up with that?

The reason is that a My Profile page is provided for every user in the portal regardless of whether they have a MySite, or even if MySites is disabled. Actually, a single page renders every MyProfile page: http://mysite/person.aspx. This page has code that displays a "Home" tab linking to the user's MySite at http://mysite/personal/userid/default.aspx if it exists. The Home tab makes it appear that these two pages are part of the same site. But the URLs hint at the truth: the My Profile page is on one site collection, while the user's MySite Home page is on another.

You knew each MySite is hosted in its own site collection, right? When a MySite is created for a user (see below), that user is added to the Owners group for the new site collection, and is also added to the Site Administrators list. This user has full power over their MySite. To customize that is a bit difficult, but possible.

I created a Visio diagram that depicts the relationships between the pages, where profile information is used on them, and how the links between them interact.

Screenshot (click to download the diagram)

What happens when you browse to http://mysite/?

Whenever you browse to http://mysite/, you're redirected to http://mysite/personal/userid/default.aspx through a few steps:

http://mysite/default.aspx inherits from Microsoft.SharePoint.Portal.WebControls.MySiteHostHomePage. This assembly authorizes the user against the MySite shared service rights (can be enabled/disabled by users or groups), then emits an HttpResponse.Redirect to either /_layouts/MySite.aspx or /_layouts/AccessDenied.aspx, accordingly.
/_layouts/MySite.aspx inherits from Microsoft.SharePoint.Portal.WebControls.CreatePersonalSpace. This assembly examines the user profile property Personal site.
- If the Personal site property is blank, CreatePersonalSpace creates a new MySite site collection for this user, then stores its relative URL (e.g. "/personal/userid/") in the Personal site property.
- CreatePersonalSpace then emits an HttpResponse.Redirect to the URL in the Personal site property.
You arrive at http://mysite/personal/userid/default.aspx, where "mysite" is the URL of the web application hosting your MySites, and "userid" is your user id.
- Note, other configurations could change this URL, such as the Personal Site Location ("personal") and the Site Naming Format (whether "userid" should include your domain.) These settings are found on the SSP Admin page "My Site settings."

Customizing MySite

Steve Peschka has the best information for customizing MySites, including how to set a custom master page automatically for all new MySites being created... this one's not obvious! Steve has a very clever approach that uploads an XML file to each new MySite, containing instructions on what should be modified. Then, a control in the custom master page reads the XML file and performs the customizations. He explains why this approach is necessary.

His XML spec for customizations provides several things you can do, such as removing/adding web parts and modifying QuickLaunch links. He doesn't provide a way to change the permissions of the owner... look for another blog posting from me on how to do that when you want to restrict the rights of each MySite's owner.

A Belated Update: the My Settings page

Because I still have a hard time finding this info, here's the explanation of why some customers find that clicking on a user name takes them to the My Site persondetails.aspx page while others only see the simple WSS userdisp.aspx page.

All people links go to http://SiteCollectionURL/SiteURL/_layouts/userdisp.aspx?ID=17&Source=... And this page decides whether to redirect to the My Site provider defined in the SSP that the Web Application containing the content site collection is assigned to based on one of the following rules:

Host name of the My Site web application matches the site from where userdisp.aspx is called (i.e. both web applications are running on different port numbers)
The site collection where userdisp.aspx is called from has a Portal Connection setting which is set to the My Site url (this would add the My Site to the start of the breadcrumb navigation for OOTB master pages which most customers don't want).

From Ian Morrish at http://www.wssdemo.com/Blog/Lists/Posts/Post.aspx?List=b853926a%2Db04e%2D4620%2D94e4%2D88a5d56cb262&ID=427&Web=d47402ad%2D1767%2D42ba%2Da072%2D133479a9bb5a. Ian describes the situation in detail, plus an unsupported fix. A supported method extends his approach to use a delegate control: http://solutionizing.net/2008/12/08/user-profile-mysite-redirect-via-delegate-control/. This references a CodePlex solution created to do this, at http://www.codeplex.com/Solutionizing/Release/ProjectReleases.aspx?ReleaseId=20923.

See also
To enable/disable MySites, Managing MySite Creation and Usage.
If MySites are disabled, Where is My Picture stored? What if MySites are off?

SharePoint 2007 Permissions Matrix

Mark Arend — Thu, 14 Feb 2008 21:05:00 GMT

This blog posting applies to Microsoft Office SharePoint Server 2007.

Many times it helps to see an overall comparison of SharePoint 2007 permissions. I assembled a matrix that shows lots of useful information that helps you understand how permission levels are related, and how they affect security trimming of Site Actions and Site Settings.

Screenshot (click to download the spreadsheet)

Update for SharePoint 2010

Colleague Rob Finney has updated this spreadsheet for 2010. Please see Rob's blog posting on the topic.