Todd's Blog

Troubleshooting SharePoint’s Page Output Caching

Mon, 07 May 2012 03:41:56 -1000

As a follow up to my previous post “When Page Output Caching Does Not Output” I have recorded a video which actually walks you through the steps and issues which I documented in this previous post. So for those of you whom don’t like to read all that much you may watch this video and/or refer back to my previous post on the same subject.

One update on this video – I mention MS11-100 and the fact it breaks page output caching in SharePoint. The SharePoint PG has released a fix which is available in the April 2012 CU. You can get more information on SharePoint CUs from my version page.

Making Debugging a SNAP

Sat, 05 May 2012 05:47:30 -1000

Recently I left Microsoft where I worked for almost 15 years and where about 10 of those years were spent in Escalation Services where my daily routine was debugging failing or faulting applications. This all began with user and kernel mode Windows processes and then once the .Net Framework shipped I move to the ASP.Net and CLR teams and began debugging more managed processes. Normally customers would send my team crash dumps or memory dumps of the offending process(s) and we would use tools such as WinDbg or CDB to dig deeper into the process to determine what was happening. There are several challenges when doing this type of work and one of the most painful is locating and referencing the correct symbols files (*.pdb). If you don’t know, symbol files are used to help give meaning to memory addresses found within a process and are most useful when building stack traces. When the debugger tools write out a process memory dump the amount of memory used by that process will equal the site of the memory dump (certain dump operation options allow you to create mini dumps but for our work a full memory dump was typically preferred). Over the years as we moved from x86 to x64 the process memory dumps being sent to us from our customers began to grow very large. It was not uncommon to have customers uploading 1-4 GB dump files (or more) and often it was not a single file that was being sent. Sometimes it was a huge challenge in capturing a memory dump at the exact time an issue was reproducing so sometimes this meant multiple memory dumps where being sent and often some of the files did not capture the event being reported. What made this a little better was the fact these files zip / rar / compress very nicely so that did help but often it was still a logistical challenge to get the correct files uploaded.

Introducing SNAP

About a year ago my friend and co-worker at Microsoft Aaron Barth and I were talking about these challenges and started brain storming ideas on how we could try to address some if not all of these challenges. Over the years we had experience with many customers whom had serious issues which only reproduced in production environments however these customers did not want, or could not, install tools or applications on production machines. From our experience most hang and performance issues could be resolved quickly or at a minimum we could be given a good line of investigation if we just knew what each thread within the process was doing at some period of time during which the issue was being experienced. We often scripted the debugger tools to create snapshots of memory dumps and once available typically dumped out each thread with a ~*kb400 command, parsed through the results and frequently where able to pinpoint the cause or develop a working theory as to the issue. But, as mentioned, installing the debugger tools, configuring scripts, and uploading one or more memory dumps hoping at least one of them was taken at the time of the event, could in some cases, take hours.

So armed with a strong motivation to make things better for our customers and heck even for us, I set off to develop a tool. I setout creating a tool called SNAP. The design goals of SNAP included:

Easy to use and deploy
No need for symbols
Generated output could be easily shared with a subject matter expert (SME) such as the developer or 3rd party support team.

The SNAP tool is very easy to use and is simply a command line tool with 3 main tasks or commands each with a number of options. The 3 commands are SNAP, DUMP, and LIST.

Snap – Captures a stack trace of the managed stack for each thread within a process and writes the output into a XML log file on disk. Each “Snap” results in an XML log file and can have the snap interval, max snap count, or duration configured.
Dump – This command captures a full memory dump of the process. It sometimes is still necessary and helpful to have a memory dump to troubleshoot some issues. Rather than having to install the debugger package we can use this tool to capture this information too. As with the Snap command this command can be configured to capture dump files on a certain interval, for a period of time or for a certain number of dump files.
List – This command is a bit like tlist.exe and will display a list of managed processes running on the current system.

Getting Started

So you are sold and want to get started using SNAP? Cool, here are the one, two, threes…

Download SNAP from the link below. The download includes a .Net 3.5 and .net 4.0 flavor of the tool along with x86 and x64 platform versions as well (so 4 exe(s) all together). Use the exe which is most appropriate for the target process you are attempting to collect information.
Select a process to “snap”. You have several choices here, you can choose the process by Process ID (PID), process name eg. w3wp.exe, or by the Application Pool Name in the case this is a IIS process. If you know you want to focus on a single process you can choose the PID option and use your favorite tool to grab the PID of the target process, I am a little partial to running “snap –c list” to get the PID but you should feel free to choose what works best for you. If you choose to use the process name and there is more than one process with this name all processes will be targetted and you will have snap logs from each – no worries because the snap log naming format will ensure you can determine which process each log is sourced.
Execute the snap command, eg “snap –c snap –p 1234“ with the snaps running wait until the issue you are troubleshooting reproduces and once complete hit Ctrl+C to stop the snap process.
Analyze the log files. I like to start with the largest file first – I open the file in a text editor and go through each stack to try determine what may be occurring at the time of the event.

To put this all together I have put together a demo video which walks you through using the Visual Studio 2010 profiler and SNAP to troubleshoot a SharePoint 2010 performance issue. If you want to just see how SNAP works for this use case then skip ahead to 14:00 into the video. While this video is focused on a SharePoint performance issue it is important to understand SNAP was not written for and has no dependencies on SharePoint so you should feel free to use this tool on any managed process.

Profiling SharePoint with Visual Studio and SNAP

Over the last year since I have written the tool it has been used across many large and mid-size customer environments to troubleshoot an entire host of issues literally around the world. Suffice it to say this tool has been well battle tested and it has proven its value in a number of scenarios where a debugger is typically needed to troubleshoot further.

I have used this tool in a number of real world farm/server down scenarios:

I have seen were a proxy was misconfigured and a majority of threads were waiting on an external web service call to return.
I have seen where a SharePoint delegate control was being used to make a reverse DNS lookup on the incoming client’s IP address and subsequently timing out – about 60% of the threads in flight were in this routine.
I have used this when requests seem to hang and noticed most threads were trying to hit SharePoint’s UPA service, which at the time the only server running that service was unavailable – again a very large majority of threads were waiting to Initialize the UPA.

In all cases all I did was analyze the largest Snap log file – look at each stack and made a determination as to what most stacks were doing at the time of the issue (see video for more details). I have not really needed to look at more than one log or have any need to create some kind of log parser or analyzer though I am sure if someone were to build one that would be well received.

As with any tool offered for free on this site I don’t offer direct support, warranty, licensing, etc. but I am willing to answer any questions you may have.

Happy Debugging!

Tool Download: http://todd.in/getsnap (681 kb)

An April Fools Prank For Nerds

Sat, 31 Mar 2012 07:06:26 -1000

So let’s say you have a sense of humor and your co-worker fails to lock his or her computer (and they have a sense of humor too [very important]). Checking out the calendar you notice it’s April first, BAM a perfect opportunity has just landed in your lap – now what? First, don’t do anything that will get you fired, because that isn’t really all that funny. So what should we do to this poor sap’s computer? Typical gags are racy desktop background images, images of a sports team that would enrage the victim, or changing their Windows theme, Hot Dog was the typical choice here – but those are old and worn. How about we take a look into the registry and see what we can jack with – Specifically let’s take a look at the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\

This key contains child keys which control execution options for processes running on the local system. Each child key is named after the process such as “myprocess.exe”. The most common control option used in Windows Vista/7 is the DisableExceptionChainValidation. So most of the child keys, which again represent a process on your machine, will have one or more values which controls the process’s execution on the local machine. There are a good number of keys which can be used to control a process’s execution but the one really interesting one to look at is named Debugger. This key is used by LSASS during the startup of a process to determine if the process should be started under a debugger. The typical use for this feature is to add this value under a key named after a process which you want to start under a debugger were the “Debugger” value is the path to the debugger you want to use, such as ntsd, cdb, or windbg. This key also opens up a couple of additional non-intended scenarios because there is no stipulation that you have to supply the path to a debugger in the “Debugger” value.

So lets try something: Under the previously mentioned key create a key named notepad.exe, add a value under that key named “Debugger” and supply the REG_SZ value “calc” (without the quotes).

With the changes in place jump to a command prompt or execute Start | Run and type in “notepad” – if you made the settings correctly you will see that the calculator started!

So what is really happening is LSASS believes calc.exe is the debugger you want to run notepad.exe under. You can see this by running Process Explorer and checking out the calculator process’s command line. As you can see calc.exe is passed a single command line option which is the path to notepad.exe. Since calc.exe does not really use command line parameters no foul is committed by passing one.

This little “trick” will not work if your faux-debugger is a process which takes in a command line option like for example mspaint.exe. So if you plan on using this in a future gag you may want to play around with a number of options on your own machine before letting loose on one of your co-workers or family members – but do not forget to undo the changes on your computer because a reboot will not undo the changes. Oh and please resist the urge to include any of the registry editing tools in your plans because you do want to undo these changes at some point.

Another Alternative Use

So I mentioned there were two uses for this trick – the second is to prevent a process from running on your system. So let’s suppose you have system administrators whom like to run various processes on your system which collect information, install other monitoring software, does a hardware or software inventory, etc. These are typically pushed to your system via Active Directory policies and while you cannot prevent that from occurring on a domain joined machine you can block these processes from running. If you can get hold of the process name, such as inventory.exe then add a key by this name under the Image File Execution Options and set the Debugger value to some process which does not exist, I like “bogus.exe”. Once complete the target process will be executed on your machine with the command line “bogus.exe c:\somepath\inventory.exe” and because the process bogus.exe does not exist on your system the process will not start and therefore neither will inventory.exe – Administrator OUT!

While this too could be used as a gag I find it more interesting to pop up a different process than nothing at all.

Happy April 1st all – Todd

VMWhere VM Manager

Tue, 06 Mar 2012 04:57:53 -1000

I have a 16GB Lenovo laptop which I use in my daily work. It runs Windows 7 and while you can install SharePoint 2010 on Windows 7 I choose never to do that (you can read more here about why I don’t use Windows 7 as my SharePoint development platform). I am not a big fan of dual, triple, quad, (or whatever comes next) booting, because as soon as I boot into one OS I will likely need to send email or do something which is setup in another OS. I also don’t like running a server OS on my laptop because I use Bluetooth every once in a while and I like the hibernate and sleep functionality Windows 7 provides. So until Windows 8 hits mainstream with its virtualization platform I must resort to running a 3rd party virtualization solution so I chose VMWare Workstation and currently I am running their latest version 8.0.

As a Microsoft employee we were strongly discouraged from running VMWare however I know a good number of consultants, whom like me, decided it was better to use the best tool for the job and ask for forgiveness if it ever came to be an issue. If asked, I never denied running VMWare but I didn’t promote VMWare or sit the box on top of my desk. Part of my daily routine was (and still is) to use Remote Desktop to access my running VMs and staying out of the VMWare Workstation console as much as possible – this kept down the questions about my loyalty and dedication to Microsoft but more importantly Remote Desktop seems faster. more responsive, and really had most if not all of the capabilities I need with a smaller memory and processor footprint. This also meant I can use the same tool to access my local VMs as I do to access other remote servers.

One of the challenges with VMWare Workstation and using Remote Desktop to access your VMs is the fact you still need to start the VMs before you can access them via Remote Desktop. This typically means you must open VMWare Workstation manager first, start the VMs, close it down with the VMs still running, and then connect with the Remote Desktop client. Out of laziness and the fact I like to code I spent a few hours and wrote a very simple tool which runs in my task tray called VMWhere VM Manager. It uses the VMWare API to start, suspend, shutdown, and turn off the guests instances running on my local machine. The VMWare API is COM based and I used a slightly modified version of the VMWare Tasks project on CodePlex to help out here. The application supports VMWare Workstation 7.0 and 8.0 running on Windows 7 with .Net Framework 3.5. The application itself is just a single EXE and DLL which can be dropped in any directory. If you want to start the application minimized you can use the “-tray” command line parameter.This is useful for when you want to start the application when Windows starts but you don’t want the window in your face.

When starting a VM from the tool you will have to wait a spell for the VM to boot and be ready for RDP access. To help I decided to hook the VMWare client tools start event and display a balloon tip to indicate the VM guest has started. As helpful as this is you will likely still need to wait a dozen or so seconds before an RDP session can be established but at least you have some indication.

Conclusion

So if you want to run your VM Guests with VMWare Workstation and use Remote Desktop to access them because it’s a better solution or because you don’t want to promote the fact you are using a non-Microsoft product VMWhere VM Manager may offer some help. The tool is totally free and as mentioned was a quick and dirty side project. As such I don’t offer support, warranty, licensing, etc. so it is a use it as you see fit but please don’t expect bi-monthly CUs.

Tool Download: VMWhere.zip (62 kb)

MS11-100 Puts the Breaks On Page Output Caching

Thu, 23 Feb 2012 04:15:39 -1000

UPDATE: The SharePoint Foundation 2010 April 2012 has a fix for this issue. Check out http://todd.in/spversions for more information.

Back in December Microsoft released a patch they called MS11-100 which addressed a vulnerability in the .Net Framework. In addition to correcting the original issue it introduced a regression which breaks SharePoint’s Page Output Caching. As mentioned in my previous post while SharePoint puts all the constructs in place for Page Output Caching its really ASP.Net which actually stores and manages the Page Output Cache on SharePoint’s behalf. As ASP.Net decides what to cache for SharePoint it looks at the HttpResponse’s Cookies collection and if any new cookies are being set/sent back to the client the page content will not be cached. As a result the next request for the same page which matches the varyby parameters set to SharePoint will result in a cache miss and the page processing again will occur.

After installing MS11-100 you may notice your cache hit rate drop to zero. If you dig a little deeper and run a Fiddler trace you may notice something odd – your SharePoint site will send back the WSS_KeepSessionAuthenticated cookie for each aspx page request even though the client already has the cookie. As you will note in the image below the same cookie is being sent from the client in the Request headers and again sent back in the Response Headers. Whenever the ASP.Net Output Caching modules sees cookies being set it will not cache the page for SharePoint.

After removing MS11-100 you may notice the cookies are no longer sent in the response and your Page Output Caching starts to work again and your Cache Hit ratio starts to climb again.

The obvious fix here is to remove MS11-100 and while that may work for some organizations its likely not going to make your security folks very happy. Another alternative is to use the bit of code below (deploying code found off the Internet is likely not to make your security folks happy either) and compile this into an assembly and add a reference into your “modules” like so:

<add name="FixSPModule" type="SharePointTrends.Fix.SpCookieFixModule, SharePointTrends.Fix" preCondition="integratedMode"/>

The module is really trivial – it basically detects if the WSS Keep Session Authenticated cookie is present in the response and in the request and removes it out of the response. This is why it is important to pay attention to the order in which you add this module into your modules list; ensure you add this entry at the bottom of all other modules, especially below the SharePoint Module.

using System;
using System.Globalization;
using System.Web;

namespace SharePointTreands.Fix
{
    public class SpCookieFixModule : IHttpModule
    {
        private const string CookieWssKeepSessionAuthenticated = "WSS_KeepSessionAuthenticated";

        public void Init(HttpApplication context)
        {
            context.PostAuthenticateRequest += PostAuthenticateRequestHandler;
        }

        private static void PostAuthenticateRequestHandler(object sender, EventArgs e)
        {
            var context = HttpContext.Current;

            if (context == null)
            {
                return;
            }

            var reqCookie = context.Request.Cookies[CookieWssKeepSessionAuthenticated];
            var respCookie = context.Response.Cookies[CookieWssKeepSessionAuthenticated];

            if (respCookie != null && reqCookie != null
                && String.Compare(reqCookie.Value, respCookie.Value, true, CultureInfo.InvariantCulture) == 0)
            {
                //we have already sent this cookie to the client
                context.Response.Cookies.Remove(CookieWssKeepSessionAuthenticated);
            }
        }

        public void Dispose()
        {
        }
    }
}

Conclusion

This regression is being reported to Microsoft today and the module provided here has been tested and confirmed to correct the issue when MS11-100 is installed.

When Page Output Caching Does Not Output

Tue, 31 Jan 2012 06:28:06 -1000

SharePoint’s Page Output Caching can offer a massive performance boost to publishing sites but only when its working and working correctly. One of the problems I have seen is when some administrators turn on Page Output Caching they just assume it works. While this may be the desire and in most cases it may just work for you I would suggest you verify; and I don’t mean hit the site with the browser to see if it speeds up.

This post is about troubleshooting SharePoint’s Page Output Caching. Now if you don’t use Page Output Caching or yours is working just fine you are the “Master of your Page Output Caching” – as for the rest of us we will likely need to put on our troubleshooting hat and dig a bit deeper. I find troubleshooting anything is allot easier if you know a little about how the component you are troubleshooting operates and how it is suppose to work.

A Little Background

SharePoint’s Page Output Caching is implemented via two HttpModules, the first being SharePoint’s own Publishing HttpModule and the second being ASP.Net’s Output Caching HttpModule - System.Web.Caching.OutputCacheModule. The two modules work together with the Publishing Module fronting all the work to determine what is available for cache and how items are to be cached based upon site collection settings, publishing page settings, and some other internal calculations which typically are the reason we end up needing troubleshooting this component (more on this later). Once the publishing cache determines the correct variables are in place to enable a page to be cached it calls into the HttpResponse.Cache.SetCaceability() which is the signal to the OutputCacheModule that we have a candidate for output caching. The PublishingModule also sets the various VaryBy parameters such as VaryByHeader, VaryByParam, and VaryByCustom which handles audience targeting, vary by browser, vary by permissions, etc. At runtime these VaryBy parameters are used to build a long string which will act as a case sensitive key which the OutputCacheModule will use to store and retrieve the contents of the page from the HttpCache which itself is stored in memory. The HttpCache is a bit like a dictionary where the key is used to lookup a value which in this case is the page contents and the key can change based upon the Page Output Cache Profile settings and the VaryBy(s).

If the Page Output Cache Profile is set to “Check for Changes” or “Perform ACL Check” then the Publishing Module will register a HttpCacheValidateHandler delegate to deal with Site Changes. When you choose “Check for Changes” every request for cached content will do a lookup to the SPSite.LastContentModifiedDate and if this date is more recent than when the item was cache the item will be invalidated and a full request is made. So any content change within a site will cause all cached pages within that site collection to be invalidated once requested – that is to say there is no “flush” operation as the documentation states but rather an item by item invalidation as the item is requested. The downside of course is for larger site collections with many sites and cached pages a single content change can cause the entire cache to become invalid – this could be a huge negative impact on performance. Pages or list items which may not have any relationship to the page requested could cause content changes which, would for a period, could cause a spike in cache misses.

Troubleshooting

Tools

So armed with our most valuable troubleshooting tool (the knowledge of how things work) it is time to look at a couple of other tools we can use. The ULS logs can be configured to log information related to Publishing Cache by jumping on over into Central Administration and enabling the Publishing Cache component’s logging. Unfortunately, while we will get a number of statements related to Output Caching we also pick up Blob Caching and Object Caching statements too. So when digging into the ULS logs be aware that some of the logging you just enabled may be related to these other caching mechanisms and not necessarily Page Output Caching.

Some of the statements you want to look for include strings like:

String	Tag
Output cache not used due to [some reason]	5kaw
Enabling specified cache profile: [profile name]	5kb0
EnableCache: [true\|false] Duration: [seconds] VaryByParam: [string] CheckForUpdates: [true\|false] CheckForRights: [true\|false]	5kb1
Invalidating URL [url to page] from the cache due to site change	921d

If you were on a desert island and could take only one Page Output Caching ULS entry with you it would have to be the one with tag 5kaw – as it is this entry which will hopefully give you a reason why your page is not being cached. I say hopefully, because as I will discuss later this is not always the case. In addition, to the strings mentioned above you will likely see a number of VaryBy related strings such as “Calculating VaryByCustom”… littered throughout the ULS logs which can be useful if your cache is growing really large and you need to determine why that may be the case.

As most of you probably already know you can request the Publishing HttpModule write a debug statement into the page which helps guide troubleshooting.

Whenever I start troubleshooting Page Output Cache issues the first thing I typically enable is the debug cache information on pages for 2 reasons, 1) I dislike trolling through the ULS logs looking for something which may not be there, the noise to signal ratio in these logs make for interesting fishing but I like my fish in water, 2) the page’s debug statement gives me almost as much information as I will find in the ULS logs. Of course one of the annoying aspects to using the caching debug statements is you must “view source” of the page in a browser and scroll to the bottom to check out the debug statement; doing this more than a few times can grow really monotonous. A little trick I use use to read the caching debug statement it to leverage tinyget.exe (yes you could use curl and a number of other tools too). I use the tool like so:

tinyget -srv:www.sp.com -uri:/pages/default.aspx -data |find "<!-- "

Using tinyget in this manner has a couple of advantages, the first being you can zero in on the debug statement, and the second is you don’t have to deal with any browser-isms as tinyget will not post any cookies, will not request any dependent requests such as style sheets, JavaScript, etc and will instead focus on and make a single page request with only the headers you supply. The downside to this tool is if you are trying to troubleshoot a problem with a specific caching profile you might not have the permissions or audience settings necessary to reproduce the issue so at this point you need to pick up your favorite fishing pole (I use ULSViewer) and cozy up to the ULS logs.

Common Issues

So the issues I typically hear the most around Page Output Caching is a page is not being cached or not being cached for the period around which is expected.

Before digging into why something is not cached we need to understand what is not inherently cached by page output caching otherwise we will be chasing a ghost. The SharePoint Publishing module ensures page requests with the POST verb and anything that lives under /_layouts will not be cached, The kicker is ASP.Net’s Output Cache module has some additional checks, such as additional verbs such as HEAD, DELETE. PUT which too are not cached.

As stated we can get reasons why pages are not cached from the ULS logs as well as with the page debug statement. The following table details these reasons

Reason	Description
The user is currently viewing the page with the console visible.	When you see this reason it typically means the page ribbon is visible and the page is in authoring mode.
Cache profile [profile name] disables caching for [Authenticated \| Anonymous] users.	If you see this message and it is not what you want you need to verify the cache profile settings.
Could not load cache profile. Profile has ID [GUID]	This is an indication someone has gone and deleted a cache profile which was in use. Since there is no way to recreate a profile’s ID it will be necessary to create a new profile and set the sites to use this profile instead of the missing cache profile.
Unable to get publishing page object for current request	This is the results of not being able to get the publishing page for for the URL passed.
Page contains personalized parts	The user making the request has at least one personalized WebPart on the page.
User has edit rights in the site collection	The user is currently viewing a page which is not in a published state and the user has Edit List Items permissions on the site (SPWeb) (not site collection as the message might indicate)
User can view an unpublished version of the current page	You might see this when “Vary by User Rights” is selected and the user has Edit List Item rights. This again, may not be an issue and you might want to check the Cache Profile settings if you see this message and don’t expect to.
There are more than 10000 uniquely secured objects in the site collection	When more than 10,000 uniquely permissioned items exist in a site collection Object Caching will just stop working. This number is hard coded and is not impacted by the Throttling settings of the web application.
The page was requested to not be cached by other code	Any User Control, WebPart, or really any other customization can explicitly request a page not be cached by calling DontEnableCachingForRequest. If you see this as a reason it is likely because of some customization calling this API.

10000 uniquely secured objects

Of the errors mentioned above the one which might leave you scratching you head is the 10,000 uniquely secured objects. Once the number of uniquely secured objects within a Site Collection hits 10,000, Object Caching just stops working – no warning, no events will be logged, just the performance of your site may drop. So do we need a health rule for this? You bet. If you happen to be lucky enough to be saddled with this reason for Page Output Caching not working, the first thing you are going to ask your self is how many are there and where are they located. For a large site collection this may be hard to determine or mitigate which means you may need to split up your site collection to help spread out the uniquely permissioned items. But before you do that you should get an idea of what they are and were they are located. We can do this with a little TSQL

SELECT ScopeUrl FROM TVF_Perms_Site('[Site ID GUID')

Before you can use this bit of TSQL you need 3 things, the target SQL server, the target database, and the Site ID. By using the URL of the site collection and this small bit of PowerShell you can get all 3.

Add-PSSnapin Microsoft.SharePoint.Powershell -EA 0

$site = Get-SPSite "http://www.sp.com"
$siteId = $site.Id
$server = $site.ContentDatabase.Server
$dbName = $site.ContentDatabase.Name

Write-Host "SQLServerName=$server DBName=$dbName ID=$siteId"

This TSQL tells us the number of uniquely secured items. If your query returns more than 10,000 rows you know you have a problem. For my demo I created a list named ListWithPermissions and added 4 items, the first 3 items I broke permissions and the 4th I left alone. I deleted item #3 and emptied it out of the recycle bin which moved it into the second level recycle bin, I then deleted item #2 and left it in the recycle bin. I then went back and modified the permissions on item #1 from what was copied from the parent permissions. As you can see from the image below we have a few things we can take away from this little demo:

Items in the recycle bin count toward our unique permission count.
Items in the 2nd level recycle bin count toward our unique permission count.
Items were the only thing we did was break permissions and copy from the parent show as unique permissions.
Items were we do not make any permission modification (item #4) do not appear in this list.

So even though my list only has two visible items with only one having unique permissions it is in fact hosting 3 uniquely permissioned items which count toward a hard coded limit of 10,000.

How you decide to fix this issue is entirely up to you but I would point out that if these types of lists are used for internal authoring or editor tracking workflow its important to understand that while the 10,000 item count might not get you the fact that any updates to these items will cause the site collection cache to be invalidated is reason enough to move them away from your content. To put it another way, if I were to edit an item in this list, a list mind you that authenticated users cannot see or use, I will effectively render my Page Output Cache invalid and cause every cached page in the Site Collection to re-render. So if you have these types of lists do not set “Check for Changes”.

Other Issues

As I mentioned earlier in this post the Publishing Module and ASP.Net’s Output Cache module work together to implement Page Output Caching for SharePoint. In most cases this works however there is one very important scenario which we need to discuss and that involves HTTP Cookies.

Lets suppose you set a page an an anonymous site to cache for 1 hour. Now normally you will see a debug cache statement such as:

What you may notice however is the date/time stamp changes more frequently than every hour. We discussed one of the reasons this may happen – content changes when “Check for Changes” is enabled. But what if this is not selected – well that leaves us with the web application is restarting, hitting another server in the load balancer with a different time stamp, or HttpCache getting trimmed. Unless you are tweaking with the web.config, deploying solutions, or running IISReset it is unlikely the web application is restarting but you might want to verify. Also if you are using a load balancer without sticky sessions you may want to eliminate this as a possible reason. With x64 systems and without making an explicit Application Pool setting you will not likely see HttpCache being trimmed due to memory pressure (if you do you have other issues you need to address first).

One scenario which everyone needs to be aware when dealing with Page Output Caching is that if a page’s response contains a cookie in its cookie collection (Response.Cookies.Add()) ASP.Net’s Output Caching module will not cache the output. So this leaves us with a debug statement and ULS log entry from SharePoint which state the page output is cached but in fact it is not – in fact you can see this by looking at ASP.Net Applications –> Output Cache Entries count in Performance Monitor. As you navigate a SharePoint site and see the “Rendered using Cache Profile…” debug statement on pages on a freshly restarted site you should see this performance counter increase. If you do not then you are likely running into this issue.

The issue tends to occur when there is some customization such as a 3rd party web analytics and/or HttpModule which uses cookies to track user activity. When these modules write new or refresh cookies using the HttpResponse’s cookie collection the page output cache will not function. But what if you don’t have any such customizations? The SharePoint product team fixed an issue with its WSS_KeepSessionAuthenticated cookie in the SharePoint 2010 December 2010 CU. This cookie was being refreshed in such a way that caused the Page Output Caching to not work properly for authenticated sites. You may want to use Fiddler’s Inspector feature to see if you have any cookies being sent back in the response headers.

Update

Wictor Wilen and I were recently discussing an issue with Page Output caching not working with one of his client’s environments. A Fiddler trace showed the WSS_KeepSessionAuthenticated cookie was being sent on every page response and as previously mentioned if cookies are sent in the response then a page will not be served from page output caching. The SharePoint farm had MS11-100 installed and once Wictor removed that patch the cookies were no longer being sent in every request and Page Output caching started working again. At this time we are not clear on why a .Net fix would impact this behavior of resending this cookie which is sourced from the SharePoint HttpModule but we did want to get it out that this patch (MS11-100) will break Page Output caching.

Conclusion

So hopefully after you read this you take away a few tips to help troubleshoot SharePoint’s Page Output Caching as well as understand its not a set and forget it feature of SharePoint – in fact I cannot think of very many that are.

Employee 45832 Has Left the Building

Wed, 09 Nov 2011 07:43:56 -1000

This Friday, 11-11-11, will mark my last day at Microsoft. After 15 years of working for the best software company in the world I have decided to accept another challenge. I have so many great memories, experiences, and friends this move was not trivial but ultimately could not be passed. I owe so much to Microsoft and all of the co-workers I have ever worked with. When my kids were in the hospital I was worried sick but I did not have worry about any bills or costs for the visits -- Microsoft’s health benefits are the best around. Having the opportunity to work with the smartest folks around only made me want to work that much harder and made me a better engineer but more importantly a better person. I have so many friends I have worked with around the world whom have all contributed to my successes and my hope is one day I will have the opportunity to payback what they have so unselfishly provided me.

I am joining an Internet startup as a partner and will be leading the design and development of our new online business to be released right around one year from now (Nov 2012). I will also work part time as an independent SharePoint consultant and will continue my role as a SharePoint MCM instructor.

So yes, my involvement with SharePoint, Microsoft, and this blog will continue for the foreseeable future and going forward I hope to see you all at SharePoint conferences around the world.

PS - yea I chose a date which could be represented in binary because that is how a true geek rolls.

Thanks
Todd Carter

SPC11 Presentations By the Numbers

Mon, 10 Oct 2011 16:49:00 -1000

A few observations after downloading the Power Point presentations from last week’s SharePoint Conference 2011

217 Power Point presentations totaling 1.12 GB
Average size for a PPT presentation is 5.5 MB
The smallest presentation had a file size of 468K
The largest presentation had a file size of about 177 MB (embedded video)
The 217 Power Point presentations had a total of 7,659 slides
There was on average 35.3 slides per presentation with a median of 33.
The most common number of slides in a presentation (mode) is 28.
There were no presentations with 46 or 55 slides.
The presentation with the most number of slides is SPC237 with 100
The presentation with the fewest slides is SPC108 with 7

…and 1 hell of a good time.

SharePoint UserInfo Table Population Tool

Fri, 23 Sep 2011 08:26:57 -1000

In a previous post I spoke about how importance of pre-populating SharePoint’s Content Database’s UserInfo table with users for landing/root (and/or very popular) sites just before a large release of a new SharePoint web application. While I did mention the API you could call to make all this happen I did not provide any tooling. This post is about a small tool I wrote, which at this point has been used with a couple of customers, to pre-populate UserInfo tables.The tool itself comes in two flavors – one for MOSS 2007 and the other for SharePoint 2010 Server. Both flavors allow you to export users from the User Profile Store to a flat file which can then be imported in a manner which populates the UserInfo table. In addition, the SharePoint 2010 version supports both Windows and Claims users.

-------------------------------------------------------------------------------
    SPUserPop SP User Population Tool version 2.0.1.0 for SharePoint 2010
-------------------------------------------------------------------------------
  Started: Friday, 23 September 2011 1:14 PM


  --Parameters--

  -o, -operation [import|export] Import/export users to/from a file from User Profile Store (Default is import)
  -a, -authtype [windows|claims] Authentication Type (Default is windows)
  -u, -url <site url> Url of site in Content DB to populate or to use as context for export from User Profile Store
  -f, -file <file path> Path to a file which contains users in DOMAIN\UserName [Windows] or Encoded Claim [Claims] format
  -w, -workers <number> the number of Import worker threads (Default 1) (Max 8)


  --Examples--

  SPUserPop -a claims -u http://team -f c:\users.txt
  SPUserPop -u http://team -f c:\users.txt -w 4
  SPUserPop -o export -u http://team -f usersFromProfileStore.txt

Export

The export operation is pretty strait forward. The command takes a URL to a site collection which will be used to gain access to the context for the source UPA or SSP to be used to to pull user accounts which are then written to a text file with each user account name on a separate line. As you would guess, any file with this format that includes usernames such as DOMAIN\Username (for Windows authentication) can be used as input for the import operation.

Import

The import operation is where the pre-population of the UserInfo table takes place. As mentioned previously the import takes a text file which includes the names of the user accounts you wish to populate. It also takes a URL to the site collection as the target to pre-populate. If you have more than one site collection you wish to populate you will need to run the tool multiple times including those site collection URLs you wish to pre-populate. For the SharePoint 2010 version of the tool there is an optionally authentication parameter which is used for when you have claims based user account names. And finally, to allow the program to run as fast as possible there is a worker parameter which will allow you to specific the number of worker threads which will handle the pre-population work. The max value for this parameter is the number of processors on the host machine * 2. Depending on the size of your SQL server, a user’s source domain and domain placement, and the machine running the tool you may want to play around with the worker threads to get the best throughput.

Update [Dec 12th, 2011]

If you plan on running this tool in a least privilege environment the user running this tool needs full control permissions to the UPSA. Being just an Administrator of the UPSA will not be enough to properly execute the tool.

Thanks to Blairb for testing this and reporting back

Download

As with every tool and source code I develop these should be considered “as is”, limited support, no warranty, but very interested in feedback.

Tool Download: SPUserPop.zip (160 kb)

Update [Sept 27, 2011]

Just received some performance numbers from a friend of mine that ran this tool at his customer’s site.

(Claims) Users exported from User Profile Store: 273,647 users in 13 minutes 35 seconds
(Claims) Users populated in SharePoint: 273,317 users in 2 hours 54 minutes 15 seconds
Number of threads used by the tool: 16 threads

Introduce Users Into the UserInfo Table

Tue, 20 Sep 2011 02:57:00 -1000

When an authenticated user, whom has never visited a site collection, first visits a site there are a number of tables within the Content database which must be updated. This activity can be expensive and performance can suffer when the site collection is the root of a web application which has just been announced or released for the first time into production. In fact, I have seen first hand this behavior take down a very large SQL server upon initial launch of a large intranet site to the point we had to roll back and I have teammates which have had the same experience (hence this post).

The SharePoint Content database’s UserInfo table stores user specific information about an authenticated user and this information and really acts as a cache which is updated periodically. When a first time user first visits a site this table must be populated. There is a row for each user and for each site collection for which they have visited. For example, if you have 100 site collections and 50 users have visited 50 of the site collections there will be 50*50 or 2,500 approximately rows in the UserInfo table.

Its not just the UserInfo table which gets updated when a new user visits a site collection for the first time but rather a whole host of tables.

UserInfo – Insert one row for new user
AllListsAux – Update the item count for the user list
AllDocs – Update the Last Modified Time and Item Count for the _catalogs/users
AllSites – Update the NextUserOrGroupdId, LastContentChanged and DiskUsed (quota management)
AllUserData – Insert a new row
EventCache – Insert two new rows into this table to track the updates to the site collection and the site collection’s root web
StorageMetrics – Insert a new row
StorageMetricsChanges – Update one row

So as you can see when a new user visits a SharePoint site collection for the first time we INSERT into 5 tables and UPDATE 4 tables. Arguably this is not a huge problem day to day but for a net new release of a really popular intranet site this can be a performance bottleneck.

The workaround for this problem is to pre-populate the UserInfo tables for site collections which host popular landing pages. Fortunately SharePoint has an API for that and its called SPWeb.EnsureUser(). This handy API takes a user’s logon name in the format of DOMAIN\Username and adds the user into the UserInfo table and the other tables I mentioned earlier. Now using this API does not mean when the user first visits there will not be any database work. While the handy dandy EnsureUser does do most of the heavy lifting, UserInfo table columns such as the tp_externalToken still need to be populated when a user first visits. Specifically here is what occurs when a user visits a site for the first time after being prepopulated:

UserInfo – Update one row for user
WebMembers – Insert with new UserID for the web in which the user just visited

As you can see when we choose to pre-populate users there is allot less work for SQL and we can push that initial load to pre-go-live activities.

So when do you use this process? I would suggest you use this process anytime both of the following are true:

A new SharePoint release with a large user population such as a new company portal or a new HR site.
A single or few landing pages within a single or few sites exist which users will either hit directly or will be auto-redirected based upon some criteria.

When planning to go with the pre-population approach its important that you identify each of the site collections which may host landing pages. Even if these pages exist across different site collections in the same content database because as I mentioned earlier, a user entry in the UserInfo table exists for each site collection the user visits. So the SPWeb to target when calling EnsureUser will be those which are hosting these landing pages.

My team and I have gone through this several times with a number of different customers and each time we have pre-populated the UserInfo table after a failed go-live we have been successful with the re-release. Conclusion; pre-populating the UserInfo table ahead of a major release of a intranet SharePoint site is a Best Practice.

Update: I have released a tool for the pre-population and you can get it from here.

Fabulous Support for the FAB40 on SharePoint 2010

Tue, 13 Sep 2011 11:03:30 -1000

As most of you will know the SharePoint Product Group (PG) released a set of templates for WSS 3.0, known as the FAB40, to assist in addressing common business scenarios. These templates for WSS 3.0 can be downloaded in a single package here. But for those of us running SharePoint 2010 Microsoft is not planning to release new versions of these templates for SharePoint Server 2010 or SharePoint Foundation 2010.

So where’s the love Microsoft? First remove the shotgun from your mouth and take a look at Samantha’s post on upgrading the FAB 40. Second, and most important, these templates are fully supported by the SharePoint PG and by Microsoft’s Commercial Technical Support (CTS) teams. This means you can open cases with CTS and request fixes for the FAB40 just as you would for any other Microsoft product which is inside its supportability lifecycle. For online support you can head on over to the TechNet Forums and post your question there.

Since Site Template files (.stp) are no longer supported you won’t just be able to apply these to a SharePoint 2010 deployment but you can upgrade sites which were sourced from these templates and you can save sites created from these templates into the new site template format, a wsp file.At this point you now have a wsp file which is based on the FAB40 template and therefore is now supported by Microsoft support.

The Lenovo W520–It’s not a W510 + 10

Mon, 15 Aug 2011 09:54:00 -1000

I have a Lenovo W510 and after reading Keith Combs’ review of the W520 I itching to get my hands on one. My new W520 arrived a couple of weeks ago – the part which I was most excited about was the new internal disk controller which moved from a SATA II to a SATA III. A long while back, just as they hit the market, I purchased a Crucial C300 which when I initially installed in my T61 cranked out some huge performance gains. I then moved it to a W510 and again saw another level of performance out of this drive. As with the other machines installing the C300 installing into the W520 I was not disappointed.

As you can see from the screenshots below I was not able to achieve more than about 280MB sec on my W510 with a SATA II controller, but on the W520 with its SATA III I pushed right at and over 350 MB sec reads. Both tests were done multiple times with Bitlocker disabled, Forefront AV was running in both cases and the latest Windows 7 x64 drivers were installed and Windows Update was all green. This performance was enough to score a 7.9 on the disk transfer rate Windows Experience Index (WEI).

For those interested, the two graphic scores are my low spot on the WEI with a 6.9 given to the Nvidia Quadro 2000M. Memory came in at 7.6 (16 GB) and the processor (Intel i7 2820QM) was 7.5.

W510	W520

We Got a Situation With The Federation

Fri, 05 Aug 2011 03:13:25 -1000

Over the last week my team has received reports from customers running both MOSS 2007 and SharePoint 2010 that Federated Search Results are no longer working. The Federated location in question seems to be centered around Bing/live.com.

I setup a test this morning in my SP2010 lab and I can confirm the out of the box (OOB) “Internet Search Results” Federated location is no longer working. In fact when you add the Federated Web Part on a Search Results page and choose this location a very long delay will occur when attempting to render the search results page. Doing a sniff I can see the original request for search.live.com gets redirected to bing.com which has been typical in the past however the difference is that instead of being an RSS response, as specified in the format query string parameter, the response is sent back in HTML which injects a delay as the results are processed. Eventually the results cannot be processed and the Search Federated Results Web Part does not render any output.

The original URL being requested is:
http://search.live.com/results.aspx?q=sharepoint&count=1&first=1&mkt=en-US&format=rss&FORM=SHAREF
which gets redirected to:
http://www.bing.com/search?q=sharepoint&count=1&first=1&mkt=en-US&format=rss&FORM=SHAREF

As you can see when you click on this link a response does come back but just not in the format we expect.

So what’s the matter you?

Well obviously need to change the Federation configuration!

UPDATE #1: Within a matter of minutes after sending an email to a Bing DL here at Microsoft I had someone pick up the phone and give me a call to discuss the issue. Within a hour I had multiple Bing team members jumping in to look at this which included reproducing the issue and discovering root cause. The Bing team has agreed to release a fix on Tuesday (Aug 9th, 2011).

UPDATE #2: As promised the Bing team deployed a fix and the world is right again. Feel free to go back into your Federation location settings and put the Query Template and More Results Template URIs back to their default values.

The fix is to change the Query Template…

FROM: http://search.live.com/results.aspx?q={searchTerms}&count={itemsPerPage}&first={startitem}&mkt={language}&format=rss&FORM=SHAREF

TO: http://api.bing.com/rss.aspx?source=web&q={searchTerms}&count={itemsPerPage}&first={startItem}&mkt={language}&FORM=SHAREF

Note I removed the format query string parameter since we are hitting a handler which returns RSS already.

In addition to changing the Query Template we want to change the “More Results” Link Template. This template is the URI which is used when someone clicks on the “More Results” link within the Search Federation Web Part. Normally this opens a new tab or browser window when clicked so we probably don’t want to display RSS but rather HTML formatted results which can be achieved using the following URI:

http://bing.com/results.aspx?q={searchTerms}&first={startItem}&mkt={language}

Once you have made these changes you need to perform an IISReset since the location configuration information is cached. After the IISReset you can test and verify the issue is resolved.Situation Out!

A Search SQL Index Issue

Tue, 05 Jul 2011 06:23:51 -1000

Ran into an issue the other day on SharePoint 2010 Search when configuring Managed Properties, specifically I was setting the MaxCharactersInPropertyStoreIndex value on a Managed Property. This is an integer value and is documented here. The maximum value for this property is 450 however if you set this value to its maximum value you will start seeing errors with your crawls and in our case the crawls never would complete. Taking a look at the application event log we can see the following error logged:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Server Search
Date:          6/26/2011 7:24:02 AM
Event ID:      57
Task Category: Search service
Level:         Warning
Keywords:
User:          DOMAIN\CRAWLACCOUNT
Computer:      COMPUTERNAME
Description:
A database error occurred. Source: Microsoft SQL Server Native Client 10.0 Code: 1946 occurred 4097 time(s) Description: Operation failed. The index entry of length 904 bytes for the index 'IX_Str' exceeds the maximum length of 900 bytes.

Taking a look at the MSSDocProps table I found the IX_Str index is defined as so:

CREATE_INDEX(MSSDocProps, IX_Str)([Pid], [strVal]) ON [PRIMARY]

The issue here is that 450 characters is actually 900 bytes because we store these characters in Unicode which is actually 2 bytes per character. Add the 4 bytes for the PID and we have 904 bytes for the index which is over the 900 byte max SQL has for index values. To work around the issue I to set the MaxCharactersInPropertyStoreIndex to 450-2 to account for the 4 bytes for the PID and this corrected the issue.

So the real maximum value for MaxCharactersInPropertyStoreIndex should be 448.

Hopefully you will never run into this issue but if you customize Search you should be aware of this limit.

The User Profile Service Heats up With a June CU Regression

Fri, 01 Jul 2011 06:56:15 -1000

Just in time for summer break here in the US the SharePoint product team has released a good number of fixes for SharePoint 2010. Service Pack 1 and the June Cumulative Update (CU) is now available for download. These are fairly substantial updates when you consider the number of bugs that have been fixed and in the case of SP1 the new improvements which have been made. To put it another way, SharePoint Server 2010 is about 1.47 GB and the Uber Package for the June CU (Foundation+Server) has a size of just about 1 GB.

In the march to continually push up the quality of SharePoint 2010 sometimes we may encounter a speed bump along the way and this is what happened to me today when installing the June 2011 CU on a SharePoint 2010 Server. This is a massive product and the sustained engineer teams have a huge task and frankly are doing a good job to keep things together. They make really tough choices every day around what gets in as a fix and what doesn’t. The team works really hard to ensure regressions do not occur however sometimes these things slip through.

My lab server was running the April CU so I installed the SharePoint Foundation SP1 and the SharePoint Server SP1 binaries and ran psconfig. I confirmed the operation completed successfully by going through the logs and using Central Administration. I then installed the June 2011 June CU Uber package and ran psconfig. I have included the text of that command here:

C:\>psconfig -cmd upgrade -inplace b2b -force -wait
SharePoint Products Configuration Wizard version 14.0.6009.1000. Copyright (C) Microsoft Corporation 2010. All rights reserved.

Performing configuration task 1 of 4
Initializing SharePoint Products upgrade...

Waiting to get a lock to upgrade the farm.

Successfully initialized SharePoint Products upgrade.

Performing configuration task 2 of 4
Initiating the upgrade sequence...

Successfully initiated the upgrade sequence.

Performing configuration task 3 of 4
Upgrading SharePoint Products...

100.00%
Failed to start service ProfileSynchronizationServiceInstance on this server after completing upgrade. Please start it manually.

Successfully upgraded SharePoint Products.

Performing configuration task 4 of 4
Finalizing the SharePoint Products configuration...

Successfully completed the SharePoint Products configuration.

Total number of configuration settings run: 4
Total number of successful configuration settings: 4
Total number of unsuccessful configuration settings: 0
Successfully stopped the configuration of SharePoint Products.
Configuration of the SharePoint Products has succeeded.

As you can see I hit a small issue. The upgrade completed however the UPS service failed to start. I confirmed as much when I took a look at my event log and found this event.

And finally, taking a look in Central Admin Services on Server I can see the User Profile Synchronization Service is “Stopped” (services.msc confirmed this too).

The error is pretty clear, the FIM service is not a fan of the DB schema on the Sync DB so I decided to reset the sync DB. Now at this point I am about to lose all my UPA settings however I cannot touch the DB and with its schema not in a good spot I really have no other choice. Fortunately we have an article on TechNet that tells you how to go about Resetting your Sync DB. http://technet.microsoft.com/en-us/library/ff681014.aspx#resetSync After running through this article I found I could still not get the UPA sync service to start. Looking in the ULS logs I found this error:

UserProfileApplication.SynchronizeMIIS: Failed to configure MIIS pre database, will attempt during next rerun. Exception: System.Configuration.ConfigurationErrorsException: ERR_INVALID_GROUPS
at Microsoft.Office.Server.UserProfiles.Synchronization.ILMPostSetupConfiguration.ValidateConfigurationResult(UInt32 result)
at Microsoft.Office.Server.UserProfiles.Synchronization.ILMPostSetupConfiguration.ConfigureMiisStage1()
at Microsoft.Office.Server.Administration.UserProfileApplication.SetupSynchronizationService(ProfileSynchronizationServiceInstance profileSyncInstance).

So clearly there is something wrong with the group membership for the UPA service. Knowing I have not changed anything in Active directory or on the machine; and knowing the TechNet article previously wanted me to add the service account (Farm Account) back to the db_owner group I suspect there was something missing in SQL. Looking back at the backup I took of my SyncDB before I blew it away I found a missing group, FIM_SynchronizationService. which my service account (Farm Account) was not a member of. So I created the group with this command and added my account to this group:

CREATE ROLE [FIM_SynchronizationService] AUTHORIZATION [dbo]

So my DB, although empty of tables, views, stored procs, and most importantly data did have a configuration that looked much like my previous DB however I just did something really bad and unsupported – I just edited the DB! The DB is empty and sans of any data so I do run some supportability risk however its probably small but I am not endorsing this approach. So after starting my UPA sync service and waiting it finally started and I was back up and read to start the configuration process again…or at least I thought…

When navigating to the UPA service to configure my synchronization connections I found I was getting a very strange error that I could not connect to SQL. So with UPA somewhat working from SharePoint’s perspective I decided to focus on FIM.

The service binary for FIM is miiserver.exe and its located in %ProgramFiles%\Microsoft Office Servers\14.0\Synchronization Service\bin. It also has a configuration file named miiserver.exe.config. Using the magic of snapshots I reverted back to before I installed the June CU and made note of the contents of this file as captured here:

Going back to my original image and taking another look at this file I can see it has changed as captured here:

See the difference? A new supportedRuntime element was added for the 4.0 version of the .Net framework.I just so happen to have the .Net 4.0 framework installed on my Windows 2008 R2 host OS so the FIM service is now attempting to use that version of the framework.

I commented out the v4 element and restarted both FIM services within services.msc. I then did an IISReset and went back into CA’s UPA configuration for sync connections and all the errors were gone. I then configured a sync connection and did a FULL sync and everything seems to be working again.

SharePoint TechFest - Dallas

Thu, 19 May 2011 13:44:08 -1000

Today I had the great privilege of speaking at the Dallas SharePoint TechFest which was held this year in the new Irving Convention Center. I did an hour long talk on SharePoint 2010 Health and Monitoring and I was asked by several folks afterwards to share out my deck. So here it is. Thanks all for a great time.

Presentation Download: SharePoint 2010 Health and Monitoring.pdf (3,331 kb)

What am I looking at here?

Thu, 19 May 2011 03:53:00 -1000

If you poke around the various icon resources included in Windows 7 DLLs you will find some really old images that have been around since the early days Windows. Some of these icons represent devices such as phones and mobile music players which are looking pretty out dated by today’s standards. According to WikiPedia the Kinect was launched in North America in November of 2010 and publicly announced June 1st 2009. The Windows 7 Beta first appeared in December 2008 and officially in January 2009. Included in the installation of Windows 7 is a DLL named mmres.dll which has a number of icon resources embedded. While digging through the icons I found one that looked familiar and eerily like a Kinect device. The screenshot below is looking at the mmres.dll from the Windows 7 Beta dated 12-12-2008. As you can see we knew what the Kinect, or Natal as it was first called, at least 6 months before it was publically announced!

Extending The SharePoint 2010 Health & Usage - Part 4: Writing a Custom Usage Receiver

Thu, 05 May 2011 05:29:32 -1000

This is the 4th article and last article in a series where I have been discussing the extensibility offered with the SharePoint 2010 Usage and Health services. If you have read all the prior articles to this point then bravo for you! I really hope this last article was worth the wait. At this point I have provided an overview of the Health and Usage Service, discussed the development of a Custom Usage Provider and showed you how to create custom Health reports and host those within Central Administration. This last article covers my favorite Health and Usage extensibility; “Usage Receivers”. In fact, here is a little secret I will let you in on, to build its analytic reports the SharePoint Web Analytics Service application uses a Usage Receiver to grab all its data from the OOB Request Usage Provider. So take a look at the SharePoint 2010 Web Analytics with all its data and rich reporting and know that all came from a Usage Receiver.

If you missed the prior articles the list below is for you, otherwise lets jump into the article…

Feature and Capability Overview
Writing a Custom Usage Provider
Writing Custom Reports
Writing a Custom Usage Receiver (this article)

Code Download: Microsoft.SP.Usage.zip (160 kb)

The Scenario

As a final reminder, I am walking you through the building a solution to track file downloads from SharePoint Document Libraries and report on the results. It’s key this solution be scalable so we will are creating a custom Usage Provider, adding a few custom usage reports in Central Administration, and we will now create a Usage Receiver to help us display download information within a Web Part for each SPSite within our environment.

Overview

A custom Usage Receiver is much like a List or List Item event receiver however you register a Usage Receiver with one or more Usage Providers. This allows your Usage Receiver to receive callbacks each time a Usage Provider completes an import of usage data. As you may recall from the first article in this series Usage Providers, are called to import data from *.usage files via a Import Timer Job which is by default set to run every 30 minutes. Once the Usage Provider has completed processing its Usage Entries any registered Usage Receivers will be called in series. If no data has been collected since the last time the Import Timer Job has fired then the Usage Provider will not be called and as a result any registered Usage Receivers will not receive a call either.

Building the Usage Receiver

If you have been following this series of articles almost every time I start building a component I always seem to start with a class which derives from some SharePoint abstract class out of Microsoft.SharePoint.Administration namespace – well this component is no different. We will start with creating a class named DownloadUsageReceiver which derives from SPUsageReciver. We only have one method we need (or even can) override which is UsageImported. When called, this method is passed a SPUsageReceiverProperties object which among other things has an IEnumerator of Usage Entries which were imported by the owning Usage Provider. There is also a reference to the Usage Provider, which was unfortunately named UsageDefinition, but I bet you could have figured that one out.

So for our solution we want to take each SPUsageEntry from the enumerator and track the total document download count and total size of all documents downloaded per Site Collection and store this total within the site collection’s root web’s AllProperties property bag. Its from here that our Web Part will pull the download information for display within the Site Collection.

This data is stored within the property bag as a string which represents a XML serialized class I call DownloadInformation. It’s a simple class which I mark as [serializable] and have two public getter/setter fields which store the data points we care about.

 [Serializable]
 public class DownloadInformation
 {
        
     public long TotalDownloads { get; set; }
     public long TotalBytesDownloaded { get; set; }

 }

Once we override the UsageImported method on our DownloadUsageReceiver class we create a dictionary of DownloadInformation objects which has a GUID key which represents the Site Collection Id. We then loop through each SPUsageEntry casting it to a DownloadUsageEntry where we then increment the total download size with the size read from the entry and update the entry count. Of course, we first have to ensure a DownloadInformation object exists in the dictionary and if that’s not the case its created.

public override void UsageImported(SPUsageReceiverProperties usageProperties)
{
    int entries = 0;
            
    try
    {

        if (usageProperties == null || usageProperties.UsageEntries == null)
        {
            return;
        }
                

        IEnumerator<SPUsageEntry> usageEntries = usageProperties.UsageEntries;


        Dictionary<Guid, DownloadInformation> siteCollectionTracking = new Dictionary<Guid, DownloadInformation>();

        while (usageEntries.MoveNext())
        {

                    
            DownloadUsageEntry entry = usageEntries.Current as DownloadUsageEntry;
            if (entry == null)
            {
                //these are not the droids we are after
                continue;
            }


            if (siteCollectionTracking.ContainsKey(entry.SiteId))
            {

                DownloadInformation downloadInfo = siteCollectionTracking[entry.SiteId];
                downloadInfo.TotalDownloads++;
                downloadInfo.TotalBytesDownloaded += entry.Size;

            }
            else
            {

                siteCollectionTracking.Add(entry.SiteId, new DownloadInformation() { TotalDownloads = 1, TotalBytesDownloaded = entry.Size });

            }

            entries++;

        }

        if (entries > 0)
        {

            PostProcess(siteCollectionTracking);
                    
        }

    }
    catch /*no throw - let the caller believe we have it under control*/
    {
        //TODO: Write this to the ULS Log, ref: http://todd-carter.com/post/2010/12/17/Yuletide-ULS.aspx
    }
            
}

Once processing has completed on each DownloadUsageEntry, and assuming we have entries, its time to call into the PostProcess method passing in the dictionary we have taken such care to create. Its within PostProcess where each Site Collection with a download is updated with new download information. Since this code is running within the OWSTimer process, which is running under the SharePoint Farm account, we should have enough permissions to each of the Site Collections – if we don’t you have bigger problems than updating download information.

So one small gotcha I want to call out; there is a potential race condition in the code below. Since each server within the SharePoint farm executes the Usage Import Timer Job independently to import data stored from within its own *.usage files its possible this code will be executed simultaneously on two different machines which both may have download information to update for the same Site Collection and its possible we might miss one of those updates by overwriting the update since we are not updating within a transaction and/or taking a farm wide lock. No worries however, its not the end of the world and I just wanted to call it out here to make you aware of the potential for this occurring.

private static void PostProcess(Dictionary<Guid, DownloadInformation> siteCollectionTracking) 
{
                
    foreach (Guid siteId in siteCollectionTracking.Keys)
    {

        using (SPSite site = new SPSite(siteId))
        {

            using (SPWeb web = site.RootWeb)
            {
                        
                bool isNew = !web.AllProperties.ContainsKey(Constants.SiteDownloadUsageKey);
                string serializedSiteDownloadInformation;
                DownloadInformation newDownloadInformation = siteCollectionTracking[siteId];
                DownloadInformation siteDownloadInformation = null;
                        

                if (!isNew)
                {
                    serializedSiteDownloadInformation = web.AllProperties[Constants.SiteDownloadUsageKey] as string;

                    if (!String.IsNullOrEmpty(serializedSiteDownloadInformation))
                    {
                        siteDownloadInformation = SerializationHelper.Deserialize<DownloadInformation>(serializedSiteDownloadInformation);

                        if (siteDownloadInformation != null)
                        {
                            siteDownloadInformation.TotalDownloads += newDownloadInformation.TotalDownloads;
                            siteDownloadInformation.TotalBytesDownloaded += newDownloadInformation.TotalBytesDownloaded;
                        }
                    }

                }
                                                
                if (siteDownloadInformation == null)
                {
                    siteDownloadInformation = newDownloadInformation;
                }

                serializedSiteDownloadInformation = SerializationHelper.Serialize<DownloadInformation>(siteDownloadInformation);

                if (isNew)
                {
                    web.AllProperties.Add(Constants.SiteDownloadUsageKey, serializedSiteDownloadInformation);
                }
                else
                {
                    web.AllProperties[Constants.SiteDownloadUsageKey] = serializedSiteDownloadInformation;
                }
                        
                web.Update();
            }

        }

    }
                                    
}

Registration

To register a Usage Receiver with a Usage Provider we need to first obtain a local reference to the Usage Provider of choice. Because each provider derives from the SPUsageProvider there is no need to cast the local reference to a specific provider type but you do want to pass the correct type into the generic GetLocal call. There is a Boolean named EnableReceivers which, when set to true, allows all registered Usage Receivers to be called. This means if you add your Usage Receiver to a Usage Provider which has this value set to false and has additional Usage Receivers configured, once you set the EnableReceivers to true so your Usage Receiver is called others may be called too. This is an unfortunate design but I don’t suspect too many folks to be impacted by this Usage Provider scoped setting.

Within a Farm scoped Feature we will call the public static methods below which are members of the DownloadUsageReceiver class.

public static void Register()
{
  var usage = SPUsageDefinition.GetLocal<DownloadUsageProvider>();
            
  if (usage != null)
  {
                
    //register the Receivers
    if (!usage.EnableReceivers)
    {
      usage.EnableReceivers = true;
    }
    usage.Receivers.Remove(typeof(DownloadUsageReceiver));
    usage.Receivers.Add(typeof(DownloadUsageReceiver));
                
    usage.Update();
  }

}

public static void Unregister()
{
  var usage = SPUsageDefinition.GetLocal<DownloadUsageProvider>();

  if (usage != null)
  {
    usage.Receivers.Remove(typeof(DownloadUsageReceiver));

    usage.Update();
  }
}

Conclusion

Think of Usage Receivers as Event Receivers for Usage Providers. As I stated at the very start of this article I consider Usage Receivers as my favorite extensibility feature of the Usage and Health service as I can think of a ton of potential and uses within my future SharePoint 2010 solutions.

I really hoped you found this series helpful but what I really hope is that I sparked some interest and you can use this in a real world SharePoint 2010 solution. If you develop such a solution I would love to hear about it – ping me via email, through my blog, or post a comment.

Happy SharePoint coding friends!

Extending The SharePoint 2010 Health & Usage - Part 3: Writing Custom Reports

Wed, 04 May 2011 11:53:49 -1000

This is the third article of a 4 part series where I discuss the extensibility of the new Health and Usage Services built into SharePoint 2010. In the second article we created a custom Usage Provider which collected download information and stored it into the Usage Database. Now its time to take a look at the data which has been collected and report on it. We will do this by extending the Health Reports which currently only include reports for Slowest Pages and Top Active Users – we should have more and we will..

Other articles in this series are as follows:

Code Download: Microsoft.SP.Usage.zip (160 kb)

The Scenario

As a reminder, I am walking you through the building a solution to track file downloads from SharePoint Document Libraries and report on the results. It’s key that this solution be scalable so we will are creating a custom usage provider, adding a few custom usage reports in Central Administration, and we will soon create a usage receiver to help us display download information within a Web Part for each SPSite within our environment.

Overview

The object of this article is to show how to create custom usage reports and expose them into the Health Reports area within SharePoint’s Central Administration.The process starts with defining the reports we would like to create, write a bit of TSQL to query the Usage DB to generate the reports, and finally register the reports with the SharePoint Health Report Store so the reports are available to users of Central Administration.

Defining Reports

For this scenario we are going to create three reports to report on our document download usage data. Each report will have a display name or title and will offer up a hopefully interesting view or pivot on the usage data we have collected. The table below documents the three reports we will create:

Report Name	Description
Top Downloaded Documents	Show the top downloaded documents by the number of downloads and include the URL, file name, extension, file size and the total size of all downloads, and the number of downloads.
Top Downloaded Extensions	Show the top downloaded documents by extension/document type. Include the average. minimum, maximum, and total download sizes as well as the total number of downloads for each extension.
Downloads By Day of Week	For each day of the week (Monday, Tuesday, etc.) show the total number of documents downloaded and the total size of all downloaded documents for the day.

Building the Query

With the reports defined lets start building our queries. The Health Reports offer a few filters which we can choose to support when building our queries which include the Server Name/machine on which the usage occurred, web application, number of items, and a date range of ‘last day’, ‘last week’, or ‘last month’. We have a choice when it comes to supporting these filters, which we will see later, but for our case we will support all of them for every report.

With Health Reports you have the option of supplying TSQL or Stored Procedures which will exist in the Usage DB and will be executed each time the report is requested. For our solution I will use a stored procedure for each of the reports. To support all the filters mentioned previously we need to use a certain pattern for the stored procedure.

CREATE PROCEDURE [dbo].[proc_{NAME}]
   @StartTime               datetime			= NULL,
   @EndTime                 datetime			= NULL,
   @WebApplicationId        uniqueIdentifier		= NULL, 
   @MachineName             nchar(128)			= NULL,
   @MaxRows                 bigint			= 100
AS
BEGIN

	/*query goes here*/

END

As you can see from the stored procedure stub above I have supplied parameters and defaults for my stored procedure which SharePoint will populate with values as the user chooses various filters. For best results I would highly recommend using this pattern as it has worked for me 100% of the time so in my mind I put that in the “proven” or “best practice” column. Now another best practice, or to put it another way, it will break if you don’t do it this way, use owner specified stored procedures and user defined functions. During the provisioning of our reports, which I will get to later, we are going to write TSQL which will provision our stored procedures and UDFs into the Usage DB. This process runs within OWSTimer and as a result runs as the SharePoint Farm Account. As you can see from the image below the default schema for this account is DOMAIN\FarmAccountName. So if you do not owner specify you will end up with a naming issue as the artifacts will be provisioned as DOMAIN\FarmAccount.StoredProcedureName which is the same issue you will run into with the UPA service which I wrote about here. DBO as the schema is not required but ensure you use the same schema name in the TSQL you use to create your stored procedures as you do when you register them with the Health Report Store.

Using the pattern I introduced previously I have written the 3 stored procedures which will be used for 3 reports which we will define. Notice that each leverage an OOB User Defined Function (UDF), fn_PartitionIdRangeMonthly which helps with the date ranges which may be passed into the stored procedure. As mentioned in the first article of this series each usage data provider has its data partitioned into 32 tables where each table represents one day of usage data based on UTC. That is, the rollover to the next table occurs at midnight UTC which is not always the local server time. This helper UDF returns a range of partition IDs for each table that fits within a supplied date range.

When writing the TSQL its important to understand the column names and column order matter and are used within the report just as they are returned. So instead of returning ‘extension’ I return ‘File Extension’ as a more friendly and meaningful name. So take care to return only the data you intend or it will wind up on your report. Keep in mind too that should you support filters the user may or may not select a filter value so you should deal with the fact the parameter may be NULL and ensure you handle that appropriately in your query.

CREATE PROCEDURE [dbo].[proc_GetTopDownloadedDocuments]
   @StartTime               datetime			= NULL,
   @EndTime                 datetime			= NULL,
   @WebApplicationId        uniqueIdentifier		= NULL, 
   @MachineName             nchar(128)			= NULL,
   @MaxRows                 bigint			= 100
AS
BEGIN

	SELECT TOP(@MaxRows) [Url], [FileName] AS [File Name], [Extension] AS [File Extension], 
		   AVG(Size) as [File Size], SUM(Size) as [Total Download Size], COUNT(RowId) AS [Downloads]	
	FROM [dbo].[DownloadUsage] WITH (READPAST)
    	WHERE  [PartitionId] IN (SELECT PartitionId FROM dbo.fn_PartitionIdRangeMonthly(@StartTime, @EndTime)) 
	AND ([LogTime] BETWEEN @StartTime AND @EndTime)
	AND (@WebApplicationId IS NULL OR WebAppId = @WebApplicationId)
 	AND (@MachineName IS NULL OR MachineName = @MachineName)
	GROUP BY [Url], [FileName], [Extension]
	ORDER BY  Count(RowId) DESC

END
GO

CREATE PROCEDURE [dbo].[proc_GetTopDownloadedExtensions]
   @StartTime               datetime			= NULL,
   @EndTime                 datetime			= NULL,
   @WebApplicationId        uniqueIdentifier		= NULL, 
   @MachineName             nchar(128)			= NULL,
   @MaxRows                 bigint			= 100
AS
BEGIN

	SELECT TOP(@MaxRows) [Extension] AS [File Extension], AVG(Size) AS [Average Size], 
		   MIN(Size) AS [Minimum Size], MAX(Size) AS [Maximum Size], SUM(Size) as [Total Download Size], COUNT(RowId) AS [Downloads]	
	FROM [dbo].[DownloadUsage] WITH (READPAST)
	WHERE  [PartitionId] IN (SELECT PartitionId FROM dbo.fn_PartitionIdRangeMonthly(@StartTime, @EndTime)) 
	AND ([LogTime] BETWEEN @StartTime AND @EndTime)
	AND (@WebApplicationId IS NULL OR WebAppId = @WebApplicationId)
	AND (@MachineName IS NULL OR MachineName = @MachineName)
	GROUP BY [Extension]
	ORDER BY  Count(RowId) DESC

END
GO

CREATE PROCEDURE [dbo].[proc_GetDownloadedByDOW]
   @StartTime               datetime			= NULL,
   @EndTime                 datetime			= NULL,
   @WebApplicationId        uniqueIdentifier		= NULL, 
   @MachineName             nchar(128)			= NULL,
   @MaxRows                 bigint			= 100
AS
BEGIN

	SELECT TOP(@MaxRows) [dbo].[GetWeekDayName]([LogTime]) as [Day of Week], SUM(Size) as [Total Download Size], COUNT(RowId) AS [Downloads]	
	FROM [dbo].[DownloadUsage] WITH (READPAST)
	WHERE [PartitionId] IN (SELECT PartitionId FROM dbo.fn_PartitionIdRangeMonthly(@StartTime, @EndTime)) 
	AND ([LogTime] BETWEEN @StartTime AND @EndTime)
	AND (@WebApplicationId IS NULL OR WebAppId = @WebApplicationId)
	AND (@MachineName IS NULL OR MachineName = @MachineName)
	GROUP BY [Extension]
	ORDER BY  Count(RowId) DESC

END

All of this TSQL is included in two different *.sql files within my solution where I have split my Install and Uninstall script. Just to be safe I have included checks during provisioning to remove any previous instance of a stored procedure before creating it again. I have marked each *.sql file’s build action with Embedded Resource in Visual Studio so it will include these files as string resources within the assembly when compiled. During my provisioning process I read from the string resource and pass this to SQL server to create or drop the stored procedures. This pattern allows me to iterate on various versions of the stored procedures from a single file without having to redeploy the solution each time.

At this point you should be doing a Different Strokes “What you talkin’ bout Willis” when I talk about creating stored procedures in the Usage DB. The Usage DB is a SharePoint database and as we all know changing the DB Schema of a SharePoint DB puts you into the realm of “The Unsupported”. The Usage DB is a little different in that its not critical path to how SharePoint operates. That is, all the data contained within the Usage DB for the most part is throw away. I have not heard of any heart ache arising from customers querying this data or using extensibility interfaces to extend the schema. This only applies to the Usage DB however so don’t try to do this for any other SharePoint DB. Since the Usage DB can grow quite large and is write optimized in most cases it makes since to move this data into a warehouse for reporting historical and trending analysis.

Report Registration

At this point its time to build the component which will register our Health Reports with the Report Store. For my implementation I created a static class named DownloadUsageReports with two public methods Register() and Unregister() which I have included below. I start the registration process by provisioning the stored procedures which, as I mentioned previously, the script is pulled from string resources embedded within the assembly. With the schema changes made I then grab a local reference to the SPHealthReportStore (found in the Microsoft.SharePoint.Administration namespace) and call into the AddHealthReport() method passing a newly constructed SPHealthReportCreationData object. The SPHealthReportCreationData is where we supply the name of the report, as it will appear in the UI, the TSQL we want to execute each time the report is executed, a Boolean indicating if the TSQL passed is a stored procedure, and then a number of additional Booleans which indicate what filters we support. For all of my reports I support everything so ‘true’ is passed all around. The report name is the primary key and as a good citizen I check to ensure the report does not already exist before attempting to add it again.

Backing out the reports is even easier and includes a call to RemoveHealthReport() passing the report name for each report I want to back out of the Health Report Store. As with almost anything SharePoint you have done nothing until you call Update() so never forget this part.

        public static void Register()
        {

            RegisterReportingStoredProcsInUsageDatabase();

            SPHealthReportStore store = SPHealthReportStore.Local;

            if (store != null)
            {
                if (!store.Contains(Constants.TopDownloadedDocReportName))
                {

                    SPHealthReportStore.Local.AddHealthReport(new SPHealthReportCreationData(Constants.TopDownloadedDocReportName,
                                                                                                Constants.TopDownloadedDocProcName,
                                                                                                true, //Is Stored Proceedure
                                                                                                true, //Support Web Application Filter
                                                                                                true, //Supports Time Filter
                                                                                                true, //Support Machine Name Filter
                                                                                                true)); //Supports Max Rows

                }


                if (!store.Contains(Constants.TopDownloadedExtReportName))
                {

                    SPHealthReportStore.Local.AddHealthReport(new SPHealthReportCreationData(Constants.TopDownloadedExtReportName,
                                                                                                Constants.TopDownloadedExtProcName,
                                                                                                true, //Is Stored Proceedure
                                                                                                true, //Support Web Application Filter
                                                                                                true, //Supports Time Filter
                                                                                                true, //Support Machine Name Filter
                                                                                                true)); //Supports Max Rows

                }


                if (!store.Contains(Constants.DownloadedByDOWReportName))
                {

                    SPHealthReportStore.Local.AddHealthReport(new SPHealthReportCreationData(Constants.DownloadedByDOWReportName,
                                                                                                Constants.DownloadedByDOWProcName,
                                                                                                true, //Is Stored Proceedure
                                                                                                true, //Support Web Application Filter
                                                                                                true, //Supports Time Filter
                                                                                                true, //Support Machine Name Filter
                                                                                                true)); //Supports Max Rows

                }


                store.Update();
            }

        }
        
        public static void Unregister()
        {

            UnregisterReportingStoredProcsInUsageDatabase();


            SPHealthReportStore store = SPHealthReportStore.Local;

            if (store != null)
            {

                store.RemoveHealthReport(Constants.TopDownloadedDocReportName);
                store.RemoveHealthReport(Constants.TopDownloadedExtReportName);
                store.RemoveHealthReport(Constants.DownloadedByDOWReportName);

                store.Update();

            }

        }

Usage Reporting

Included below are a few screen shots with each representing the reports we defined earlier.

Unfortunately we cannot control the column width or supply any additional formatting on pre-render so things like the Total Download Size will be written as 32488646 rather than 324,88,646. Now of course you could do some formatting within your stored procedure however keep in mind that each column in this report is sortable so any formatting you do which may change the column sort is just going to confuse your users more than if you just left the formatting as is. Another option is to create application pages and deploy those into Central Administration and use additional reporting techniques such as visualization by leveraging OOB SharePoint chart controls or 3rd party controls (ComponentArt controls were used for the screenshot below).

Conclusion

As you have seen creating reports for your Usage data is relatively trivial. The tax you pay for this ease however is the limited formatting you can apply to the Health Reports. The reports we show here however are only available to Central Administration users and our end users cannot take advantage of this information. Since we do not want to open our Usage DB up to direct queries outside of Central Administration but we may want to expose a subset of this data to our user base we need to extend the solution further. In the next, and last, article in this series I will show you how to create a Usage Receiver which we can leverage to expose some usage data to end users.

Extending The SharePoint 2010 Health & Usage - Part 2: Writing a Custom Usage Provider

Tue, 03 May 2011 04:42:55 -1000

This is the second article in a 4 part series were I discuss the new Health & Usage Services built into SharePoint 2010 and how they can be extended to build some very interesting solutions. In this article I will discuss the process around creating a custom usage provider, dive into some internals, and provide code examples which will pull it all together.

Other articles in this series are as follows:

Feature and Capability Overview
Writing a Custom Usage Provider (this article)
Writing Custom Reports
Writing a Custom Usage Receiver

Code Download: Microsoft.SP.Usage.zip (160 kb)

The Scenario

As a reminder, I will walk you through the building a solution to track file downloads from SharePoint Document Libraries and report on the results. Its key that this solution be scalable so we will create a custom usage provider, add a few custom usage reports in Central Administration, and we will create a usage receiver to help us display download information within a Web Part for each SPSite within our environment.

Overview

As mentioned in the first article, a Usage Provider can be used to collect and store additional usage information in either the Usage Database or a custom data store. Its typical to use a custom usage provider when there is usage occurring which you would like to track or additional details for a usage event which is not being collected for which you would like to collect. These two scenarios are important because there is a ton of usage data being collected already by the OOB providers and it does not make since to collect the same data twice. For example, lets say you wanted to track aspx page requests; while you could write a custom provider to track these requests it’s a bit silly since the RequestUsage provider collects and logs this information into the Usage Database and its trivial to parse through the data to get a view of the aspx requests. For our scenario we want to track downloads of documents from document libraries and while much of the information we are going to collect is already available in the RequestUsage provider we do have a couple key parameter(s) missing; the List ID and fact the List ID represents a document library. While the URL is available its error prone to parse the URL in an attempt to glean which requests are for document downloads.

This solution calls for 3 primary components, the Usage Entry, the Usage Definition, and the component which will detect the download and log the fact it occurred. The Usage Entry and Usage Definition will use base classes from the SharePoint namespace Microsoft.SharePoint.Administration while the actual component which does the logging will be implemented as a custom Asp.Net HttpModule. Our provider will be simple and strait forward (because that is how I role) as we will leverage the Usage Database as our central store and we will not require any additional processing. A standard Visual Studio 2010 SharePoint project is our starting point where we will create a Farm solution which will deploy our single assembly into the GAC. We will wind up with a Farm scoped Feature which will register our custom Usage Provider as well as Web Application scoped feature which will register our custom logger HttpModule into the associated web application’s web.config using the much loved and hated SPWebConfigModification API.

Usage Entry

The Usage entry is the class which will be used to store the parameters we want to collect and record for each document download. This first step naturally is to decide which parameters we wish to collect and give them proper names. The key thing to remember about this step is that everything you decide to collect will be stored in the Usage Database (should you choose to store your data here and not another custom data store). The table below lists out the parameters I intend to collect for each download usage event. As you can see I have the SQL and .Net type listed as well as the column names which will be used within SQL. We will use the SQL specific values later in our custom Usage Provider but its good to understand the translation at this point.

Column Name	Column Type	Description
WebAppId	uniqueidentifier/Guid	Web Application ID
SiteId	uniqueidentifier/Guid	Site Collection ID
WebId	uniqueidentifier/Guid	Site ID
ListId	uniqueidentifier/Guid	List ID of the Document Library
ItemId	uniqueidentifier/Guid	Item ID of the file being downloaded
Url	nvarchar(256)/String	Full URL
FileName	nvarchar(256)/String	file name of the file being downloaded
Extension	nvarchar(256)/String	extension of the file being downloaded
Size	bigint/long	the size of the downloaded file
UserAddress	nvarchar(46)/String	the source/client IP address

So given the table above you may be thinking, why are we not logging the user name of the user who made the request/download.Well the short answer is, we are. For each usage entry a number of default parameters/columns are collected for us automatically. As you can see from the table below, which lists all default OOB parameters, the UserLogin will take care of collecting the user name for us so there is no need to duplicate effort here.

Column Name	Column Type	Description
LogTime	datetime	The time the actual event occurred. This is different from RowCreatedTime as that value represents the time which the event was logged into the database.
MachineName	nvarchar(128)	The machine which is recording this event
FarmId	uniqueidentifier	The Farm ID
SiteSubscriptionId	uniqueidentifier	For multi tenant environments this is the tenant ID
UserLogin	nvarchar(300)	The user performing the request
CorrelationId	uniqueidentifier	The Correlation ID which is unique identifier for the usage request.

So now that we know the data we want to collect its time to start coding the usage entry. This class, which I have named DownloadUsageEntry, derives from the SharePoint abstract class SPUsageEntry and we need to mark the class with the [Serializeable] attribute. Our Usage Entry class also needs to implement the ISerializableUsageEntry interface which uses a key/value pair type of serialization scheme through a specific constructor and through the GetObjectData method.

namespace Microsoft.SharePoint.Administration
{
    public interface ISerializableUsageEntry
    {
        void GetObjectData(SPKeyValuePairSerializationInfo info);
    }
}

The DownloadUsageEntry class will actually not have all that much code since its primary a storage mechanism. In fact most of the code is around implementing the ISerializableUsageEntry interface. In the sample solution download you can open the DownloadUsageEntry.cs and take a look at the pattern that I used to implement this interface, for me, like any good pattern, its easy to understand, repeatable, and above all, its correct.

So here is the constructor and deserialization:

        internal DownloadUsageEntry(SPKeyValuePairSerializationInfo info) : base(info)
        {
            
            info.TryGetValue<Guid>(Constants.WebAppIdColumnName, out _webAppId);
            info.TryGetValue<Guid>(Constants.SiteIdColumnName, out _siteId);
            info.TryGetValue<Guid>(Constants.WebIdColumnName, out _webId);
            info.TryGetValue<Guid>(Constants.ListIdColumnName, out _listId);
            info.TryGetValue<Guid>(Constants.ItemIdColumnName, out _itemId);
            info.TryGetValue<String>(Constants.UrlColumnName, out _url);
            info.TryGetValue<String>(Constants.FileNameColumnName, out _fileName);
            info.TryGetValue<String>(Constants.ExtensionColumnName, out _extension);
            info.TryGetValue<long>(Constants.SizeColumnName, out _size);
            info.TryGetValue<String>(Constants.UserAddressColumnName, out _userAddress);

        }

And here is the serialization:

        void ISerializableUsageEntry.GetObjectData(SPKeyValuePairSerializationInfo info)
        {
            
            info.AddValue(Constants.WebAppIdColumnName, _webAppId);
            info.AddValue(Constants.SiteIdColumnName, _siteId);
            info.AddValue(Constants.WebIdColumnName, _webId);
            info.AddValue(Constants.ListIdColumnName, _listId);
            info.AddValue(Constants.ItemIdColumnName, _itemId);
            info.AddValue(Constants.UrlColumnName, _url);
            info.AddValue(Constants.FileNameColumnName, _fileName);
            info.AddValue(Constants.ExtensionColumnName, _extension);
            info.AddValue(Constants.SizeColumnName, _size);
            info.AddValue(Constants.UserAddressColumnName, _userAddress);
            
        }

The public fields:

        public Guid WebAppId { get { return _webAppId; } set { _webAppId = value; } }
        public Guid SiteId { get { return _siteId; } set { _siteId = value; } }
        public Guid WebId { get { return _webId; } set { _webId = value; } }
        public Guid ListId { get { return _listId; } set { _listId = value; } }
        public Guid ItemId { get { return _itemId; } set { _itemId = value; } }
        public string Url { get { return _url; } set { _url = value; } }
        public string FileName { get { return _fileName; } set { _fileName = value; } }
        public string Extension { get { return _extension; } set { _extension = value; } }
        public long Size { get { return _size; } set { _size = value; } }
        public string UserAddress { get { return _userAddress; } set { _userAddress = value; } }

You may have noticed the pattern I chose to use does not use the field shortcut syntax such as: public Guid WebAppId { get; set; }, instead I have a private member variable backing the field. The reason for this is because in the constructor the method to get the values out of the SPKeyValuePairSerializationInfo object takes an ‘out’ parameter were the actual value is used and a field reference cannot be passed as an out parameter.

In addition to the interface implementation the class should override the ParentType field to return the .net type of the custom Usage Provider we will write later. The last piece of this class is the fields we will use for the parameters which will be used by the HttpModule to pass the values collected for a usage event into our class local variables.

Usage Definition

The Usage Definition is the component which acts are our traffic cop or manager for the Provider. Its here were we define our SQL Table and columns which will be provisioned within the Usage DB. We start by creating a class that derives from SharePoint abstract SPUsageProvider and overrides the TableName, Columns, UsageEntryType, and Description fields supplying our own values.

    public sealed class DownloadUsageProvider : SPUsageProvider
    {
        private static SPColumnDefinition[] s_columns = new SPColumnDefinition[]
        {
            new SPColumnDefinition(Constants.WebAppIdColumnName, SqlDbType.UniqueIdentifier),
            new SPColumnDefinition(Constants.SiteIdColumnName, SqlDbType.UniqueIdentifier),
            new SPColumnDefinition(Constants.WebIdColumnName, SqlDbType.UniqueIdentifier),
            new SPColumnDefinition(Constants.ListIdColumnName, SqlDbType.UniqueIdentifier),
            new SPColumnDefinition(Constants.ItemIdColumnName, SqlDbType.UniqueIdentifier),
            new SPColumnDefinition(Constants.UrlColumnName, SqlDbType.NVarChar, 256),
            new SPColumnDefinition(Constants.FileNameColumnName, SqlDbType.NVarChar, 256),
            new SPColumnDefinition(Constants.ExtensionColumnName, SqlDbType.NVarChar, 256),
            new SPColumnDefinition(Constants.SizeColumnName, SqlDbType.BigInt),
            new SPColumnDefinition(Constants.UserAddressColumnName, SqlDbType.NVarChar, 46)
        };
        
        public DownloadUsageProvider() { }

        public DownloadUsageProvider(SPFarm farm) : base(farm) { }

        public override string TableName
        {
            get { return Constants.DownloadUsageProviderName; }
        }
        
        public override IList<SPColumnDefinition> Columns
        {
            get { return s_columns.ToList(); }
        }

        public override Type UsageEntryType
        {
            get { return typeof(DownloadUsageEntry); }
        }

        public override string Description
        {
            get { return Constants.DownloadUsageProviderDescription; }
        }
        
        public static DownloadUsageProvider Local
        {
            get { return SPUsageDefinition.GetLocal<DownloadUsageProvider>(); }
        }
        
        public static void Register()
        {
            var usage = DownloadUsageProvider.Local;
            if (usage != null && usage.Enabled == false)
            {
                usage.Enabled = true;
                usage.Update();
            }
        }

        public static void Unregister()
        {
            var usage = DownloadUsageProvider.Local;
            if (usage != null && usage.Enabled == true)
            {
                usage.Enabled = false;
                usage.Update();
                usage.Delete();
            }
        }
    }

The TableName and Description fields are just strings which I just pull out of a static constants class. The table name for this example is DownloadUsage and we do not need to owner specify the name, eg do not prefix with a ‘dbo’. We use and array of SPColumnDefinition objects to define each column of information from our Usage Entry class. The SPColumnDefinition defines the name of the column and its SQL type. The class has a couple of constructors, one which takes the column name and SQL type, and another constructor which takes an additional parameter which you must use when the SQL Type is a string type such as SqlDBType.NVarChar and this parameter defines the length of the string. There is a one to one mapping of SPColumnDefinition to the columns which will be created within the Usage DB for our provider tables and order within the array matters as that will be the order within the table. In addition, as mentioned previously, there are a number of OOB columns which we get for free. The columns are hosted within 32 tables which are created for our custom provider automatically. The tables sport the name of our provider followed by _partitionx where X is the number designator (see Part 1 for more information). The Usage Manager is also kind enough to create a SQL view which encompasses the 32 tables, as well as it creates two stored procedures, prc_Enum{provider-name} and prc_Insert{provider-name}, as well as a Tabled Value Function. The stored procedures are used by the Usage Manager to write the Usage Information into the SQL Tables so we do not use these directly ourselves.

Since the Download Usage provider code supplied is a fairly simple solution you may be wondering, if you so chose, how you can leverage an external data store. What you do not see in the code above is several more overrides (which I will stub out for you in the code download accompanying this series). There are 4 overrides or hooks into the Usage data processing pipeline which you may choose to override.

ImportEntries – This method is called just as usage entries are ready to be imported. It takes a enumerator or SPUsageEntry objects which you can choose to cast to your Usage Entry class and do any additional processing or manipulation before being written into the data store.
WriteDataRowToServer – This method is passed in the table name, columns, log time, machine name, and row information for each usage entry. Should you choose to store usage information outside of the Usage DB its this method which you will want to override to control the redirection of where the usage information will be stored.
ProcessData – This method is called once a day by the Usage Daily Timer Job and takes no input parameters. The purpose of this method is to do any daily processing of usage entries such as aggregation or additional processing which may be required by your provider.
TruncateData – This method too is called once a day by the Usage Daily Timer job and again takes no input parameters. The purpose of this method is to do any daily truncating, cleaning, or archiving of Usage entries from a custom data store.

One last point on the ProcessData and TruncateDate methods, if you are developing a custom solution which uses this methods you may be thinking debugging these methods is as simple as executing the Usage Daily Timer Job. For the ProcessData you would be correct, this method is called each time this job is executed and so if you execute the job multiple times within a single day to debug your custom provider this process will work. The TruncateData is a bit different however and does a check within the Usage Daily Timer Job which ensures it will only call TruncateData once per calendar day (as defined by the server’s time zone). Keep this in mind if you are waiting on your TruncateData breakpoint to hit, it may take a while.

Back to the code -- the Local field and Register/Unregister methods are not required by this class but are, as you would guess, for registration and un-registration of the provider. I simply added them as statics to this class but you could locate them anywhere within your solution. The methods themselves are pretty self explanatory and we simply just set the Enabled field on a local instance of our Provider and call Update(). Once the provider is registered you may notice the SQL schema objects have not been created; don’t freak out – this is not a bug, these objects will be created on the next run of the Usage Data Import timer job assuming there is Usage information recorded for this provider. There is a bug however when you unregister a custom usage provider were the Usage DB schema is not cleaned of the tables, view, stored procedures, etc. This is not the case for Health Providers as they seem to work fine because of a call into the stored procedure, prc_CleanObjectsHelper, which cleans out the SQL Schema. Usage Providers just don’t call this stored procedure with SharePoint 2010 RTM. Manually calling prc_CleanObjectsHelper passing the name of your provider will clean out each table, view, and stored procedure created for your provider however it leaves behind a single User Defined Function (UDF) which you can drop for yourself if needed. This may be the case if you are developing a custom usage provider and you need to iterate on your Usage Entry/Definition design. Just be aware that if the DB is not clean when you attempt to instantiate a custom usage provider the process may fail or you will not see the new definition represented in the DB until you clean out the DB Schema. Its my understanding this is on the plate to be fixed if not already fixed in one of the later CUs.

Logging the Download Event

With our custom Download Usage Provider defined its time to start looking at how we actually record a document download. While not specifically part of the custom Usage Providers its this part of the code which actually makes the Usage Provider dance and its probably the most challenging. While SharePoint does provide a variety of hooks such as list, web, and workflow event handlers is often the case the usage you are interested in capturing does not have an associated event handler. For example, I would love to have crawl event handlers such as: CrawlStarted, CrawlStarting, CrawlCompleted, ItemCrawled, but I regress. For our scenario there is not an ItemDownloaded event handler on a SharePoint list so we will need to write a custom HttpModule to capture the download event. There are 3 download scenarios we need to capture and subsequently log:

User clicks on a document in a library and downloads
User opens a document in Office Web Applications (OWA) and then choose the “Open in Word” or “Open in PowerPoint” link.
User chooses “send to” -> ”Download a copy” from the item control menu

I won’t go into all the details around how I detect the various ways a user may download a document from a document library but the code will all be included in the solution download (once available). What I will talk about is the part of the HttpModule were we log into our custom Usage Provider. The code below is a snippet from the HttpModule which logs the Usage event. We first get a reference to the SPUsageManager which we will call LogUsage() passing our custom Usage Entry object, DownloadUsageEntry. As you can see we need to populate each of the fields with the data we wish to collect. Note OOB properties such as the user name, correlation ID, and machine name are auto collected by the base SPUsageEntry class so we do not need to supply those values.

            //make is so...
            
            bool result = false;

            try
            {
                
                SPUsageManager usageManager = SPUsageManager.Local;
                
                if (usageManager != null)
                {

                    DownloadUsageEntry entry = new DownloadUsageEntry
                    {
                       
                        WebAppId = SPContext.Current.Web.Site.WebApplication.Id,
                        SiteId = SPContext.Current.Web.Site.ID,
                        WebId = SPContext.Current.Web.ID,
                        ListId = library.ID,
                        ItemId = file.Item.UniqueId,
                        Url = context.Request.Url.ToString(),
                        UserAddress = context.Request.UserHostAddress,
                        Size = file.Length,
                        FileName = file.Name,
                        Extension = Path.GetExtension(file.Name)
                    };

                    result = usageManager.LogUsage(entry);


                }
                

            }
            catch (ThreadAbortException) { throw; }
            catch {/*Best Effort*/}

Configuration

For our Download Usage Provider there are not many knobs to turn when it comes to configuration. We have the ability to turn off or on the provider but that is really about all there is unless you want to tweak the 2 Usage Timer jobs which control the importing of the usage data into the Usage DB and the daily Timer job which does aggregation; more information on these Timer Jobs can be found in Part 1. Custom Usage Providers which use external data stores outside of the Usage Database may have additional configuration needs which will need to be addressed specifically for each such provider.

Conclusion

As you can see its really easy to build a custom SharePoint 2010 Usage Provider (how many things in SharePoint can you say that about). But the real money shot is leveraging the data collected with the provider within custom solutions and reporting. In the next article in this series I will show you how to create custom reports on the data which is being collected. While I am off working on that article I will leave you with a simple screen shot of a query I did to get the top 10 downloaded documents for the month of April. Trivial, right!

Grabbing Your Pretty Mug from Active Directory

Mon, 02 May 2011 08:23:43 -1000

I do not normally create blog posts around topics which are covered on other blogs on the Internet however this topic seems to still draw a bit of confusion as well as I wanted to offer a very complete picture (pun totally intended) of how to move user pictures from Active Directory into SharePoint User Profiles.

Getting Started

For this walkthrough I am using a SharePoint 2010 Farm running the April 2011 CU and a single domain controller on Windows 2008 R2.

Domain: SPCOM
DNS Domain: sp.com
My Site Host: http://my

I have a number of demo users I have created within Active Directory and I have imported pictures/photos for each user via PowerShell. The PowerShell snip below takes a path to a jpg file ($fileName), reads the file into a byte array, and puts the data into the thumbnailPhoto attribute of a user.

$temp = get-content -encoding byte $fileName 
$photo =  [byte[]]$temp 

$newUser.put("thumbnailPhoto", $photo)
$newuser.SetInfo()

Once the photo has been set for the user we can use ADSI Edit to confirm as much:

Configure SharePoint

At this point we are ready to configure SharePoint 2010. From the User Profile administration pages select Manage User Properties and choose the edit the Picture property. Scroll to the bottom of this page and choose or ensure the Source Data Connection is the connection to your Active Directory. If you don’t have anything in this drop down list it means you have not configured User Profile Synchronization. I suggest you stop right here and jump on over to Spence’s blog post on the topic. For the attribute choose thunbnailPhoto and direction should be Import. At this point please do yourself a favor and hit the Add button. A common mistake is to skip over the add button and hit the OK button however you are going to be disappointed with the results because that effectively does nothing for you and makes no changes to the property mapping.

Before we move on lets confirm the mapping is in place.From the properties list we should see the picture property with a mapped attribute of thumbnailPhoto as in the image below.

Jumping back into the properties of the Picture attribute we should see the mapping here too. If we have these settings in place we are good to move on.

With the User Properties configured property its time to start a full profile sync. Its important this is a full sync and not an incremental. It’s a good bet the profile sync is going to take a fair amount of time.

Once the full profile sync is complete we need to run a bit of PowerShell which will migrate the user pictures from the User Profile store to a location where they can be accessed via a URL and used within SharePoint. This PowerShell was added to the product in the October 2010 CU and the command is documented here. The command is super simple to run and the PowerShell snip below shows you what this looks like for my environment.

Add-PSSnapin Microsoft.SharePoint.PowerShell -erroraction SilentlyContinue

### CreateThumbnailsForImportedPhotos added in Oct 2010 CU ####
Update-SPProfilePhotoStore -MySiteHostLocation http://my -CreateThumbnailsForImportedPhotos $true
####

There are a couple of things you need to be aware of before you attempt to run this command:

The user running this command will need permissions to execute PowerShell against the User Profile Manager. To ensure this is the case open Central Admin, navigate to Manage Service Applications, highlight the User Profile Synchronization Service Service Application and choose Permissions. Note I did not mention Administrators but rather Permissions. From inside the Permissions dialog add the user which whom you will use to run the PowerShell script. If you don’t do this you will receive an error such as: Update-SPProfilePhotoStore : Object reference not set to an instance of an object and as they say in Texas, that dog won’t hunt.
This command walks through your entire User Profile store pulling out each image for every user and creates a large, medium and small thumbnail and uploads those images into the User Photos picture library at the root Site Collection of your My Site Host. So for 30,000 users you will end up with about 90,000 images in this library. The PowerShell command will pause or block until this is complete so just plan for this to take a while.

Once you have finished running the CreateThumbnailsForImportedPhotos command (or while waiting for it to complete), you can open the My Site Host, choose View All Site Content, and verify the number of items in the User Photos picture library.

That’s it, you are done; you have just successfully migrated pictures from Active Directory into the SharePoint User Profile store, and then finally into the My Site host for your farm.

Extending The SharePoint 2010 Health & Usage - Part 1: Feature and Capability Overview

Tue, 26 Apr 2011 08:19:56 -1000

This is the first article in a 4 part series were I discuss the Health & Usage Services built into SharePoint 2010 and how they can be extended to build some very interesting solutions. My hope is that after reading the series you find really cool ways to use the Usage Services and I hope to hear back on what you all have built.

The 4 part series breaks down as so with all parts will be published in quick succession so I won’t keep you waiting for the ending.

Feature and Capability Overview (this article)
Writing a Custom Usage Provider
Writing Custom Reports
Writing a Custom Usage Receiver

Code Download: Microsoft.SP.Usage.zip (160 kb)

Overview

In this first article we need to lay some groundwork before digging into code. The SharePoint 2010 Usage and Health Data Collection component is new with SharePoint 2010 and ships as part of the Foundation SKU so everything we will be discussing throughout the series is equally applicable to any SharePoint SKU. The service is comprised of a single Service Application and database, Health Providers, Usage Providers, and a number of SharePoint Timer jobs.

There are two distinct types of data which are collected and reported on with this service; usage and health data.The usage data can be used to gain insight into what your users, services, and other components within SharePoint are accessing or what actions they may be performing. For example, one of the most useful bits of usage information collected is the Request Usage which catalogs each request received from users by SharePoint. This data is a bit like the traditional IIS logs in that much of the same data is collected however unlike IIS logs additional SharePoint specific information is included with each request which provides a better context of SharePoint intrinsic objects which were involved in the request. For example, the Web Application ID, Site ID, Web ID, and correlation IDs are collected for each request which allow the administrator to gain a deeper understanding of usage for each Site and Web as well as correlation of errors conditions, which typically display or log correlation IDs, back to the request which produced the error. In addition to the usage information you may choose to have health data collected which too can be a valuable tool when troubleshooting error conditions or performance issues. An administration may choose to have SharePoint ULS, Windows Event Logs, Performance counters, and SQL DVM information collected. All of this information is collected through one of two provider interfaces, Usage Providers and Health Providers. The way to distinguish between these providers is to think about Usage Provider data as any information which is collected as a result of some usage activity such as a user request or execution of a service. The more users generating request the more usage data which will be collected. Health Provider data is collected regardless of any usage and is outside the scope of any user activity, for example, performance monitor information can be continually collected even when there is no requests being made of the system.

Usage and Health Data Collection Service Application

Its important to note the Usage and Health Data Collection Service Application is not a “normal” Service Application in that it does not appear in the UI where we might see other Service Applications. This service application can only be installed once per farm and cannot be consumed across farms. In my previous post, where I provide a simple PowerShell script to get started deploying Service Applications, you may notice I create the Usage and Health Data Collection Service Application as the first Service Application. This is not be accident, some Service Applications such as the SharePoint Search Service Application require an instance of the Usage and Health Data Collection Service Application to exist, so when provisioning the SharePoint Search Service Application if one does not exist it will be created with default settings which may not be desired. No worries if this has happened to you already or if you might have leaned on the Farm Configuration Wizard for a bit of assistance – a little additional PowerShell can re-created your Usage Database with the name and on the SQL instance you desire.

Usage and Health Database

The Usage and Health Data Collection Service Application’s main data store is the Usage Database. As mentioned previously we only get one Health Data Collection Service Application per SharePoint farm which can contain a single reference to a SQL Database so if you plan to make heavy use of this database it may make since to host it on a dedicated LUN with plenty of space and potentially even a dedicated SQL instance. The DB itself is write optimized and stores Usage information such as page requests, search query usage, timer job usage, etc. It also stores health information such as Event logs, ULS logs, and performance counter data, etc. In most cases it does not make since to attempt to mirror or configure this DB for log shipping as in most cases this data can be considered throw away – its usage and health data, your users will create more!

By default the retention period for the Usage Database data is 14 days however an administrator can change this to up to 31 days if desired. There is one gotcha you need to be aware of, anytime the retention period is modified any usage data which may exist at the time of the change will be lost.

As you would expect the data is stored within the database in tables. Each Usage and Health provider has its own set of tables with which to store its collected data. The tables are partitioned, not using SQL partitioning, but rather by creating 32 tables for each provider. The tables use a naming schema such as: dbo.{provider-name}_Partition{0-31}, where the provider-name is replaced by the name of the provider and the number designator of 0-31 is used to distinguish between each of the tables. Provider data is written into each table based upon UTC time and for each day a new table is used. So while each table holds a days worth of collected data, unless your servers are in the GMT time zone you will have to look across 2 tables to get a view of a single day’s worth of information. No worries however because included with each provider is a SQL view named dbo.{provider-name} which can be used to join the 32 partitioned tables into a single view of all your data.

Since I have been discussing the database schema and providing information on were you can find all this great diagnostic information its probably a good time to discuss the database access policy as it relates to the health and usage DB. As you may know the SharePoint support team (and the product team for that matter) have a very strict policy when it comes to modifying and querying databases which are owned by SharePoint. There are obviously very reasonable reasons why these rules exist and you are best to abide by them. The Usage DB is a bit different and you can query and extend the schema of this DB without getting into trouble from support as long as you add your own objects and don’t try to modify OOB. It’s still advisable however that if you plan to do some very heavy querying that you move this data off onto another DB and optionally another SQL instance otherwise you could impact the ability to capture the very information you are after. An example of where you may want to extend the schema with custom stored procedures or UDFs is provided in the part 3 of this series were we build custom reports.

Data Movement

The Health providers capture data and log directly into the Usage and Health Database however Usage information takes a bit of a different path. Remember usage data collection is associated with system requests and if we were to attempt to log into SQL each time one of these events were raised it would not be long before we would surly see performance issues. To ensure performance is not an issue the SPUsageManager uses ETW tracing to write the collected usage information into binary “*.usage” files on the local machine. The SPUsageManager wraps the ETW tracing and uses binary serialization to push an instance of a SPUsageEntry object into the usage files (I will discuss how you create your own SPUsageEntry classes in the next article) for each usage request which is logged.

Two timer jobs are employed to move and process usage data. The first timer job is the Microsoft SharePoint Foundation Usage Data Import, this job runs every 30 minutes and picks up any data which was written into the *.usage files and moves these into the Usage database. It does this by reading the *.usage files and calling the Usage Provider which will write the information into the Usage DB. After each Usage Provider has had the opportunity to process its data any registered Usage Receivers will be called which optionally may do additional processing. Think of Usage Receivers much like SPList event recovers where the usage information collected for a provider will be passed to the receiver each time the Microsoft SharePoint Foundation Usage Data Import service runs - assuming there is data for the Usage Provider for which the Usage Receiver is registered. In part 4 of this series I will cover creating a custom Usage Receiver in more detail.

Usage Providers don’t have to write their data into the Usage DB however all of the OOB Usage providers do. In the figure above you will note the ability for a custom Usage Provider to either write to the Usage DB or another data store. You may also notice that custom Usage Receivers can not only listen for Usage import events from custom usage providers but also from the OOB Usage Providers. In fact, this is how the SharePoint Web Analytics Service Application receives its data. The SharePoint Web Analytics Service Application registers a custom Usage Receiver to the OOB SPRequestUsageDefinition, aka, the Request Usage provider, and uses this receiver to move data into its staging database for further processing to be used for the analytic reporting.

The second and final timer job is the Microsoft SharePoint Foundation Usage Data Processing which runs daily and calls into each Usage Provider allowing the provider to do any daily aggregation processing (ProcessData()) and truncating of data (TruncateData()). This processing is optional and not all OOB providers leverage this process. In fact the only OOB provider that does take advantage of this interface and uses the ProcessData() method to process/update usage information for each Site Collection, eg the SPSite.UsageInfo.

Extensibility

Like so many components within SharePoint the Health and Monitoring functionality can be extended in a number of ways, for example:

Health Rules – Useful for periodic checks against rules
Usage Providers & Usage Entries (Part 2) – Collect usage related data and import/process into data store
Usage Reports (Part 3) – add additional usage reports
Usage Receivers (Part 4) – Receive notifications of OOB or custom usage definition data imports for additional processing and/or reporting.
Diagnostics Providers – Collect non-usage data and import/process into data store.

The Scenario

For the remaining posts in this series I will walk you through the building a solution to track file downloads from SharePoint Document Libraries and report on the results. Its key that this solution be scalable so we will create a custom usage provider, add a few custom usage reports into Central Administration, and we will create a usage receiver to help us show download information for each SPSite within our environment.

Update a List Item Without Anyone Looking

Thu, 21 Apr 2011 11:38:11 -1000

So lets say you want to change a list item but you don’t want to impact any versioning or having to deal with check-in or check-out, and you don’t want to change the last modified or modified by values. You can do this with the bit of code shown below. Grab the item reference and update the item just as you normally would however rather than calling SPItem.Update() you want to call SPItem.SystemUpdate(). You can check out the TechNet reference if you want to see additional code samples.

using (SPSite site = new SPSite("http://team"))
{
  SPList list = site.RootWeb.Lists.TryGetList("Annoucements");
  if (list != null)
  {
    SPListItem item = list.Items[0];
    item["Title"] = DateTime.Now.ToLongTimeString();
    item.SystemUpdate(false);
  }
}

There are two variations of this method and the one I called here passes false to indicate that I don’t want to increment the version count of the object for which I am updating, useful if you have versioning enabled on the list and ignored if you don’t. The second variation of the method takes no parameters and is the same as calling SystemUpdate(true) which does increment the version on the item.

So after executing the code above here is what my Announcements list looks like. Note the lack of change of the Modified date and the fact the title was set to the current date/time. Keep in mind too that although you made this change and it appears to be transparent to the list the auditing logs (if enabled) will pick up the change just as if you had called the Update() method so this is not a way to get around auditing.

What Sites do I have permissions to?

Tue, 12 Apr 2011 02:32:48 -1000

Just had a conversation with a couple instructors in the speaker room here at the BPUK Conference. What is the most efficient way to get a list of sites a user has permissions to. Search of course, it does the security trimming for you and it has a master list of all sites within SharePoint (assuming you crawl SharePoint sites).

The trick here is to format a Keyword query search with the proper properties, in this case it’s the contentclass property with a value of sts_site. Optionally we might want to add additional constructs which would limit the scope of the search such as using the PATH property and using suffix matching.

So here is a query logged on as the Administrator…

And here is another logged on as Dan Bacon Jr…

We Don’t Need No Stinking Patches

Fri, 18 Mar 2011 03:56:56 -1000

Keeping up with WSS, MOSS, SharePoint Foundation, and SharePoint patches can be a challenge. The guys on my team work with many different clients whom typically run at different SharePoint versions and patch levels. Its often we have only version numbers rather than the specific patch or KB which documents an installed patch. In addition, we sometimes have the need to build out lab environments at the same patch level as the client. Currently we use a crib sheet which documents each patch and version and from time to time makes its way around the team via email. Rather than continue to send out updated versions of this document to the team DL we wanted to make it available in one place and to everyone. The SharePoint version list will be stored at this location: http://todd-carter.com/page/SharePointVersions.aspx I have added a navigation link at the top of this blog to this location as well in case you cannot remember the URL. I have modified the original list by added links to the KB articles and the download locations when available and I hope you find this list useful.

Its important to note this list was originally compiled by Eric Adams, Jose Vigenor, and Brett Smith so when you see these guys at a conference buy them a beer.

SSD Disk Allocation Size Test

Fri, 28 Jan 2011 03:28:15 -1000

Once every couple of years or so Microsoft provides me with a new laptop. This time around it was a Lenovo W510 with a quad proc (8x if you count Hyper-Threading) with 16 GB of RAM. Although the machine came with a standard 7200 RPM drive I already had two SSD drives which I plopped into the machine within minutes. Because I was moving data around and wanted to optimize things if I could on disk I wanted to look at the allocation size I used to format the SSD drives to see if I could not eek out any additional performance. Using the secondary drive as my test I formatted the OCZ 120 GB drive using the minimum, default, and maximum allocation sizes available.

I then used the ATTO Disk Benchmark tool to test each configuration. As you can see there is really no major deviations between the 3 however the 512 byte allocation size doesn’t kick as much on the low to mid end. As for the larger allocation size I saw nothing that was compelling enough to make that move especially given the slack space tax I would be paying for that configuration when smaller files are stored on disk. So in the end I went with the default allocation size for my SSD drives.

512 bytes (min)	4096 bytes (default)	64k bytes (max)

Yuletide ULS

Fri, 17 Dec 2010 00:24:37 -1000

In this season of giving why not consider adding ULS logging to your customizations for your SharePoint administrators. When troubleshooting a critical issue around the holidays nothing goes better with the old Eggnog and Jack Daniels than some good old fashion, the way your mama did it, diagnostic logging. In this post I will show you a pattern that I use which not only provides an easy way to add that logging to your SharePoint customizations, e.g.. one method call, but also allows the SharePoint Administrator, tat has to keep your code up and running, the ability to customize and manage the Tracing and Event levels from within Central Administrator just like any other logging component within SharePoint Server 2010.

Solution

The full solution is available for download at the end of this post.

For our scenario I am going to build ULS logging into a project named “Level Zero”. The project has several components including Pages, Web Parts, etc and we need to supply an easy to use interface for developers to write trace and event messages to. The entire solution here is only two classes. The first class is the LevelZeroService which derives from SharePoint’s SPDiagnosticsServiceBase class. When logging to the ULS Areas and Categories are used to segment entries between components and sub-components. This is an especially nice feature when using a tool such as ULS Viewer to filter through the logs. My implementation only exposes one Area however more could be added within the appropriately named method ProvideAreas where I return an IEnumerable set of SPDiagnosticsArea instances. For each area there is a set of one or more Categories which are of type SPDiagnosticsCategory. It’s the category were you supply a default Trace and Event level. These default levels are used when no custom logging levels have been configured within Central Administration (more on this later). I used Monitorable and Error as the default Trace and Event levels respectively because this is what the out of the box (OOTB) SharePoint categories use and I didn’t want to be different. When writing trace or event messages you must access the Area’s Categories collection by the category name. Since I did not want to expose an interface that relied on me to remember the various category names or how to case and spell them I created a Category enum which I use to reference the categories in a type safe way. Take note that whatever order the categories are added to the SPDiagnosticsArea is the order in which they are displayed to the administrator within the diagnostics logging page.

public sealed class LevelZeroService : SPDiagnosticsServiceBase 
{ 
  private const string AreaName = "Level Zero";

  public enum Category
  {
      Email,
      General,
      Modules,
      Pages,
      SiteManagement,
      WebParts,
      Workflow
  };
           

  public LevelZeroService() 
      : this("LevelZeroDiagnostics", SPFarm.Local)
  {
  }

  public LevelZeroService(string name, SPFarm farm)
      : base(name, farm)
  {
  }

  internal static LevelZeroService Local 
  { 
      get 
      {
          return SPDiagnosticsServiceBase.GetLocal<LevelZeroService>();
      } 
  } 
     
  protected override IEnumerable<SPDiagnosticsArea> ProvideAreas() 
  { 
      
      List<SPDiagnosticsCategory> categories = new List<SPDiagnosticsCategory>();

      foreach (string categoryName in Enum.GetNames(typeof(Category)))
      {
          uint categoryId = (uint)(int)Enum.Parse(typeof(Category), categoryName);

          categories.Add(new SPDiagnosticsCategory(   categoryName, 
                                                      TraceSeverity.Monitorable, 
                                                      EventSeverity.Error, 
                                                      0, 
                                                      categoryId));
      }

      yield return new SPDiagnosticsArea(AreaName, categories);
  }

  internal SPDiagnosticsCategory this[string categoryName]
  {
      get
      {
          return Areas[AreaName].Categories[categoryName];
      }
  }
}

With the service out of the way now I can create the static Diagnostics class which exposes a number of static methods such as Trace, WriteEvent, and WriteException. These public methods simply call into the internal interface on my service class. Below is the Trace method which as you can see gets a reference to the service and calls the WriteTrace method, or in the case of the the Events, calls WriteEvent. The Diagnostics class also has a WriteDebug method which I added as a failsafe for when I need to get logging quickly and just want to use DbgView.

public static void Trace(uint tagId, LevelZeroService.Category category, TraceSeverity severity, string message, params object[] data)
{
  LevelZeroService localService = null;

  try
  {
    localService = LevelZeroService.Local;
  }
  catch (Exception ex)
  {
    Debug.WriteLine(String.Format("ERROR: Failed to log Trace message: {0}", ex.Message));
  }
  
  if (localService != null)
  {
    localService.WriteTrace(tagId, localService[category.ToString()], severity, message, data);
  }

  WriteDebug(category, severity.ToString(), message, data);

}

Those astute developers will notice that I do a try/catch and just swallow any exception which may be thrown. I do this for a couple of reasons, first, the call to the Local property seems to be the only place where I could possibly through and exception in my testing so I needed to log something so as to help figure out why logging may not be occurring, and second I didn’t want to re-throw the exception because its likely that for whatever reason it is that I cannot get a reference to the service its not about to clear up anytime soon and throwing and error here may mean the developer calling this method might catch that exception and attempt to log it too, basically you end up with a Chicken/Egg issue if you don’t just swallow any exceptions here.

Its important to note that almost all customizations you write within SharePoint no matter the process they may run in can leverage the ULS logging mechanism, even console applications running on the server can write events. The exception to this is a custom SharePoint Search connector. This assembly will run within the SharePoint Search daemon process which does not write into the ULS so the failsafe WriteDebug method I mentioned previously will be needed when logging is required in this scenario.

Deployment

Once you download the solution you will find there are two projects, a SharePoint project which includes the source code we have discussed up to this point and a simple console application to test the functionality. Unlike most SharePoint projects this one does not have any Features defined. Its not really necessary since all we are really doing is placing our assembly in the GAC. Oh didn’t I mention that, yea the assembly needs to be strong named and installed into the GAC. So why didn’t I just create a class assembly project in Visual Studio 2010? Good question, for at least 2 reasons: 1) When a SharePoint solution is deployed within Central Administration its deployed across all servers in the farm auto-magically, 2) All of my customizations are going to take a dependency on this project and therefore I can add an activation dependency within my dependent SharePoint projects to ensure the diagnostic package is in place.

Once the solution has been added and deployed within SharePoint we now need to make a simple registry addition on each server within the farm (this does not include SQL Servers) to register our service with SharePoint. In the download I included a setup.cmd file which make the registry change which simply writes the class name and 4 part assembly name of our service into a location where SharePoint can locate it.

reg Add "hklm\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\WSS\Services\[ServiceName]" /v AssemblyQualifiedName /d "[class name + 4 part assembly name]" /f

Once the registry change has been made on all servers its now time to run PSConfig to install the service.

PSCONFIG.EXE -cmd services –install

This command should be run this from just one serer in the farm (I like the machine hosting CA myself). It should be noted its always a best practice when you do anything with psconfig that you do this during non peak or off hours.

Now its time to navigate to Central Administration (CA) and take a look at the diagnostic logging. As you can see from the image below our new area and its child categories are nestled all snug in their beds right amongst SharePoint’s out of the box settings just waiting for our administrators to configure further.

The Test

With deployment complete its time to test the solution. Included in the download is a test project which is just a simple console application which when executed calls into the Diagnostics class’s public methods and logs several Trace and Event messages into the ULS log.

    class Program
    {
        static void Main(string[] args)
        {

            Diagnostics.Trace(  LevelZeroService.Category.General, 
                                TraceSeverity.Unexpected, 
                                "This was unexpected so I added logging");

            Diagnostics.Trace(  LevelZeroService.Category.General, 
                                TraceSeverity.Unexpected, 
                                "This was unexpected too, it happened at {0}", 
                                DateTime.Now);

            Diagnostics.WriteEvent( LevelZeroService.Category.Pages, 
                                    EventSeverity.ErrorCritical, 
                                    "Call the developer at home.");

            Diagnostics.WriteException( LevelZeroService.Category.WebParts, 
                                        new SPException("A better error message than we get most of the time..."));
            
        }
    }

So if everything has gone correct at this point we can open up our trusty notepad and point it to our ULS logs directory and see the impact we have made to our administrator’s lives with the small bit of code we have added. As you can see both the Trace and Event calls have made it into our ULS logs.

Using my favorite ULS looky tool ULSViewer I am able to filter on the events from my test tool. ULS Viewer has a thing against the name “Area” and instead calls it “Product” but it’s the same field so just do the substitution in your head

Since we also logged a couple of super critical Events these not only where written to the ULS but to the event log as well. So hopefully we have some monitoring happening at the event logs that we can hook up to our new fancy logging mechanism.

At this point you are ready to wrap up this functionality, put a nice bow on it, and include it in your projects. I am no St. Nick but trust me, your administrators will love you for providing such a wonder gift that keeps on giving.

Microsoft.SP.Diagnostics.zip (45 kb)

How Many Versions Do You Have?

Tue, 14 Dec 2010 02:49:19 -1000

Seems like a simple question however for SharePoint 2007 and 2010 it may be harder than you think to get a solid answer. What is really behind the question is how much space is my versions taking up or to put it another way if I trim my versions how much space will I get back. Unfortunately we have hit upon two shortcomings with SharePoint, 1) there is no out of the box reporting mechanism to give a SharePoint administrator any insight into number of versions or storage and 2) there is no out of the box trimming policy or mechanism which can be applied. For the most part we are left with implementing storage quota and either increasing quota when requested or asking users to do a self trim of documents and/or versions manually.

Later in this post I have included two TSQL scripts you can run against each of your Content DBs to get a better handle on document versions. The scripts use the NOLOCK construct but I would encourage not running them against a live production system and backups of your content DBs work nicely here – oh you don’t have backups of your content DBs, well then your version issue is only temporary. :-)

The scripts focus on two aspects of version policy which I have seen incorporated – keep only x versions of a document and keep all versions for x years. After running the scripts below and through the magic of Excel 2010 I produced the following two reports. The first is document versions based upon age where age is by year. The way you read this thus, for YearOfAge of 0 means versions that are less than a year old, YearOfAge of 1 is versions with an age of 1 year or more but less than 2, etc. The SizeByYear and DocCountByYear represent the size and number of documents for that particular year. The TotalDocCount and TotalSize is a sum of the current and prior year values. For example, there are 265,748 documents which are more than 1 year old with a total size of 102,763,586,583 bytes. So if someone was to make an argument to delete all versions 2 years of age or older then we can see that will clean up about 33,615 documents and about 16 GB of space.

The second report is based on number and size of documents by a specific version. Same data points as above but this time we are focusing on versions rather than a time span with SizeByVersion and DocCountByVersion being specific to a single version. TotalSize and TotalDocCount are a sum of prior versions. So its correct to say there are 304,260 documents with 7 or more versions with a combined size of 85,978,585,323 bytes.

So using these two TSQL scripts you can answer the version question. So how do I clean up those versions based upon some time aspect or number of versions or a combination of both? This will be in a later blog post.

TSQL scripts

As with anything on this blog these are not supported but supplied as “sample code”. I am not a SQL guy so those that sleep TSQL will undoubtedly point out my deficiencies and I welcome any improvements you may offer.

SharePoint 2010

WITH VersionAgeTable
AS
(
SELECT  DATEDIFF(year, [TimeCreated], GETUTCDATE()) AS [YearsOfAge], 
		SUM(CAST([Size] AS BIGINT)) AS [SizeByYear], 
		COUNT(*) AS [DocCountByYear]
FROM [dbo].[AllDocVersions] WITH (NOLOCK)
GROUP BY DATEDIFF(year, [TimeCreated], GETUTCDATE())
)
SELECT  d.[YearsOfAge],
		d.[SizeByYear] AS [SizeByYear],
		(SELECT SUM([SizeByYear]) FROM VersionAgeTable WHERE [YearsOfAge] >= d.[YearsOfAge]) AS [TotalSize],
		d.[DocCountByYear] AS [DocCountByYear],
		(SELECT SUM([DocCountByYear]) FROM VersionAgeTable WHERE [YearsOfAge] >= d.[YearsOfAge]) AS [TotalDocCount]
FROM VersionAgeTable AS d
ORDER BY d.[YearsOfAge];


WITH VersionTable
AS
(
	SELECT  [InternalVersion]/512 AS [Version], 
			SUM(CAST([Size] AS BIGINT)) AS [SizeByVersion], 
		    COUNT(*) AS [DocCountByVersion]
	FROM [dbo].[AllDocVersions] WITH (NOLOCK)
	GROUP BY [InternalVersion]/512
)
SELECT  d.[Version],
		d.[SizeByVersion] AS [SizeByVersion],
		(SELECT SUM([SizeByVersion]) FROM VersionTable WHERE [Version] >= d.[Version]) AS [TotalSize],
		d.[DocCountByVersion] AS [DocCountByVersion],
		(SELECT SUM([DocCountByVersion]) FROM VersionTable WHERE [Version] >= d.[Version]) AS [TotalDocCount]
FROM VersionTable AS d
ORDER BY d.[Version];

SharePoint 2007

WITH VersionAgeTable
AS
(
SELECT  DATEDIFF(year, [TimeCreated], GETUTCDATE()) AS [YearsOfAge], 
		SUM(CAST([Size] AS BIGINT)) AS [SizeByYear], 
		COUNT(*) AS [DocCountByYear]
FROM [dbo].[AllDocVersions] WITH (NOLOCK)
GROUP BY DATEDIFF(year, [TimeCreated], GETUTCDATE())
)
SELECT  d.[YearsOfAge],
		d.[SizeByYear] AS [SizeByYear],
		(SELECT SUM([SizeByYear]) FROM VersionAgeTable WHERE [YearsOfAge] >= d.[YearsOfAge]) AS [TotalSize],
		d.[DocCountByYear] AS [DocCountByYear],
		(SELECT SUM([DocCountByYear]) FROM VersionAgeTable WHERE [YearsOfAge] >= d.[YearsOfAge]) AS [TotalDocCount]
FROM VersionAgeTable AS d
ORDER BY d.[YearsOfAge];


WITH VersionTable
AS
(
	SELECT  [Version]/512 AS [Version], 
			SUM(CAST([Size] AS BIGINT)) AS [SizeByVersion], 
		    COUNT(*) AS [DocCountByVersion]
	FROM [dbo].[AllDocVersions] WITH (NOLOCK)
	GROUP BY [Version]/512
)
SELECT  d.[Version],
		d.[SizeByVersion] AS [SizeByVersion],
		(SELECT SUM([SizeByVersion]) FROM VersionTable WHERE [Version] >= d.[Version]) AS [TotalSize],
		d.[DocCountByVersion] AS [DocCountByVersion],
		(SELECT SUM([DocCountByVersion]) FROM VersionTable WHERE [Version] >= d.[Version]) AS [TotalDocCount]
FROM VersionTable AS d
ORDER BY d.[Version];

With Managed Metadata You have a Choice

Mon, 06 Dec 2010 05:36:16 -1000

In this blog post I am going to show how to use an alternate rendering mechanism to render Managed Metadata Fields which allows them to behave like standard Choice Fields.

A Brief Background

Managed Metadata fields are a really cool feature of SharePoint 2010. They allow an administrator to configure a field in SharePoint to source its values from one or more terms within a term set. Terms can be used in a number of ways such as tagging and in fields within lists and libraries. I have had several of my customers want to move away from using Choice based fields for some of their content types to Managed Metadata fields however the editing experience changes. Choice fields usually render as either a dropdown list or in the case of where multi values are a set of checkboxes for each potential choice value. Managed Metadata Field editing is done with a single textbox which leverages AJAX to implement type ahead auto population. As a result a user can start typing part of the name of a term and a list will pop within the UI to help filter the choice of terms. There is also a button next to the textbox which allows the user to popup a modal dialog that displays the choice of terms. As you can probably tell by this description the user experience can be quite different when moving from a SharePoint Choice Field to a Managed Metadata Field and users will most likely find this move at least a little confusing.

So out of the box why can’t we render Managed Metadata fields the same way we render Choice fields? Rendering Managed Metadata Fields as a set of checkboxes or as a drop down list does have its limits such as:

Terms can have sub-Terms which do not render easily with Drop Down or checkbox controls.
A Term Set can contain a great number of Terms to choose from and the Managed Metadata Select dialog is going to better handle this large set of choices than a drop down list. In addition this dialog will better handle the case when multi values are allowed vs. a large number of checkbox controls.
The default rendering of Manage Metadata Fields allows for new Terms to be added in the case where the field is exposing an open Term Set for which the user has permissions to add Terms.
With the Office 2010 Client there is parity for editing Managed Metadata fields with the out of the box rendering on the server.

So given all these limits I think you can see why the decision was made to go with the implementation we have today. That said, there are still use cases where a Choice Field type of rendering mechanism will work and in this blog I will show you one way to accomplish this and as I go through the scenario I will point out advantages of this approach.

The Scenario

Lets suppose we have a retailer which uses SharePoint to host a list of shirts where each shirt has a single Primary Color and zero or more Secondary Colors.

Today the list has two independent choice fields for the color fields and when a change needs to occur such as adding a new color or removing a color from the list of choices this requires changes to the content type which must be pushed down to ever list which may leverage the color scheme. Moving forward we may want to use a Term Set with Terms for each color so we have a single source for where we can manage, add, remove, and deprecate colors without having to edit and redeploy the content type.

To start I created a Term Set named “Colors” and loaded a number of Terms, each representing a color I want as a choice for the fields for which the Term Set will be used. Now at this point I can choose to create two Managed Metadata Fields and hook them up to the “Color” Term Set and offer my users a relatively compelling experience. However in my scenario my Term Set is fairly small and it’s a closed Term Set, that is, I will own adding, updating, and deleting of Terms from my Term Set and not allowing users to contribute their own colors. The image on the left shows my Term Set and notice I have marked one term “Teal” as deprecated which in my custom field type which I will detail later is ignored as a valid choice. The image below shows what my list’s edit dialog may look like when using the out of the box Managed Metadata rendering.

The Project

Before digging into Visual Studio we need to understand what we are about to create. A Field in SharePoint has really 4 main components, a Field Type, Field Value, Field Control, and finally a Field Definition. In chapter 10 of Andrew Connell’s book SharePoint 2007 Web Content Management Development on page 158 he has a really nice image which puts these elements together which I will attempt to explain here:

Field Type - This is the main entry point class which implements the custom field. Your custom Field Type will typically derive from one of the out of the box Field Types such as SPFieldLookup, SPFieldUrl, etc which all each derive from the base SPField. The primary job of the Field Type is to expose the Field Value and Field Control to callers.
Field Value - This class wraps the actual value or values being stored. There is no SharePoint inheritance base class structure which you need to follow or any interface which you need to implement. Its all up to you on what you want to store here.
Field Control – This class needs to derive from BaseFieldControl and is responsible for the rendering of the control when being edited by the user.
Field Definition – Unlike the previous items this is an XML file that describes to SharePoint the field type and its properties and default rendering.

So now that we understand the components lets talk about how we are going to implement each. As mentioned the Field Control handles the edit time rendering and therefore the main bulk of our effort will be spent here changing what is to be exposed to users. The class will derive from BaseFieldControl and will handle the case when the field handles single or multi-value selections. Next is Field Type, the Managed Metadata column uses a class named TaxonomyField and for the most part does everything we need from this type minus one small detail. Since it’s the class which exposes the Field Control we need to create a new Field Type which derives from TaxonomyField but exposes our custom Field Control. Since we are not changing what is stored for the Managed Metadata Field we do not need to change the Field Type implementation which is implemented for a single choice field as TaxonomyFieldValue and for a mutli-choice as TaxonomyFieldValueCollection.

To get started we need to create a Field Definition which I named “Managed Metadata Choice”. I sourced this definition from the Managed Metadata Field Definition stored in fldtypes_taxonomy.xml which is located under the 14 hive under TEMPLATE\XML\. The XML needs to be modified to point to the assembly and class which is to represent our new custom Field Type, in this case TaxonomyChoiceFieldType.

The project, which is available for download below, is a relatively simple SharePoint project in that is only contains a few classes, one XML definition file and does not have any Features. One thing to be aware of is that after you deploy this solution you will need to do an IIS Reset before SharePoint will pick up the new custom Field.

This solution has a couple of limitations, first it does not allow the user to add additional terms when using an open term set. It also does not allow a user to choose sub-Terms. For example, if you had a Term Set named “Shapes” and sub-Terms “Square” and “Triangle” and chose to use the root of the Term Set as the anchor the only Term rendered would be “Shapes” and the sub-Terms would not be exposed.

The Result

The dialog below shows what this looks like after the custom field has been deployed and new columns instantiated with the custom field. The Primary Color field is a single select so its rendered as a drop down much in line with how Choice Fields are rendered. The Secondary Colors Field is build off our custom Field type and the option to allow multi-select was chosen therefore its rendered as a set of checkbox controls. What is not clear from this image is the fact that if you choose to localized your Term Set these two fields would be localized too which is a clear advantage over standard Choice Field types.

Conclusion

While normally the default Taxonomy Field rendering will work for most scenarios there are cases where it may make since to offer an alternative editing experience for Managed Metadata fields which closely resembles that of the Choice Field type especially when replacing a Choice Field with a Managed Metadata Field.

Microsoft.SP.Taxonomy.zip (83 kb)

Don’t Tell SharePoint The Name of Your SQL Server

Fri, 03 Dec 2010 04:59:06 -1000

SQL Aliases have been around for a while however I am amazed at how many folks either don’t know about them or don’t use them. A SQL Alias is just a really easy way to do a string substitution of a connection string just before a connection is made to a SQL server. The SQL Alias configuration is stored in the registry and the anywhere the server name portion of the connection string matches the server alias the substation is made.

Lets look at a quick example, suppose you have a web application you are working with on your laptop which makes a connection to your local SQL instance. You store the SQL connection strings in the web.config but when you deploy the application to a QA or production server you will most likely not be connecting back to the SQL instance on your laptop. This is a great scenario for SQL Alias, choose an alias name, in this example I chose TX-DB-A as my alias and my local dev machine is TX-DEV-01. Once created I then update the connection string in my web.config to use TX-DB-A as my server name.

In production, where I am to deploy my web application project I need to create a similar alias however this time it references a different server name.

At this point I can deploy my web application from my laptop to production without having to remember to edit the web.config.

Note in these two examples I used a naming format of [Location]-[Environment]-[Designator] where the servers are in Texas (TX) and DEV and PROD represent my development and production environments respectively. the Designator is the unique identifier and for real servers I use a number but for SQL Alias I use an alpha character such as A, B, C, etc. The reason for this I that in any error logs or when looking at any configurations and trying to troubleshoot connection issues I need to determine if what I am seeing is a SQL alias or a real server. For example, I cannot ping TX-DB-A, and in production if I see errors connecting to TX-DB-A I need to check on a different server than I would if I noticed those same errors in my development environment.

Spence did a good write up on why you really need SQL Alias to get around a SharePoint bug however I personally always recommend SQL Alias – even if you never use it it’s a level of indirection which costs you nothing but allows flexibility when needed. And the good part is its really easy to test the failover to another SQL instance if needed. The downside is the SQL Alias is machine wide and you cannot point the alias for some DBs to one server and others to another, unless you already broke out your Alias before your configured SharePoint.

So consider this pattern – for each new DB you add to SharePoint create a 1:1 SQL alias.

DBName Alias SQLServer

Content_DB_1 SQL-CONTENT-DB1 sql1.contoso.com

Content_DB_2 SQL-CONTENT-DB2 sql1.contoso.com

Content_DB_3 SQL-CONTENT-DB3 sql1.contoso.com

Content_DB_4 SQL-CONTENT-DB4 sql1.contoso.com

Now you can move Content DBs between servers with little impact.

1. Set Content DB 1 to Read only

2. Take backup of Content DB 1

3. Restore Content DB 1 on sql2.contoso.com

4. Update SQL alias on all servers

5. Set Content DB to read/write

Now this obviously does not scale when you are talking about hundreds of DBs but you get my point. MySite Content DBs might use a different SQL Alias than Intranet or Team Content DBs, etc.

Transform Your Search Results

Wed, 06 Oct 2010 01:51:00 -1000

In a previous blog post I talked about how you can use XSLT to display additional Managed properties returned from Search. In this post I want to talk about the method I went about to create the XSLT in the hopes that you find it an interesting enough pattern to use should you find yourself creating some of the same visualizations for Search, the CQWP, Lists, or the BCS WebParts which all use XSL to transform XML into HTML.

The process of writing XSL transforms is a bit like an algebra equation where you are solving for the unknown. For example, you are moving through a pipeline of sorts XML –> XSLT –> HTML where the unknown is the XSLT which you need to develop. As in algebra this is a heck of allot easier when you are solving for just one unknown, so this means to get started you need to nail down the other two variables, the XML and the HTML.

The HTML can be mocked up to represent the finished product and then shopped around to get by in from your stakeholders to ensure you are producing the results they require before getting started. Once that is set you need to get your hands on the XML which will be the source data you will begin to refactor into your mockup HTML. The source of the XML in this example will be from Search as a response from a query. The easiest way I have found to get a sample of XML from Search is to edit the properties of the Core Results WebPart and replace the default XSLT with the XSL below. This XSL just simply passes through the XML directly onto the page which is hosting the webpart (in our case the Core Results WebPart).

<?xml version="1.0" encoding="utf-8"?>
<!-- THIS XSLT Gets the XML document passed in and returns it-->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt" exclude-result-prefixes="msxsl">
    <xsl:output method="xml" indent="yes"/>
    <xsl:template match="@* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()"/>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

Saving this configuration and supplying a sample search will result in an ugly looking page. Open that page’s source by doing a “view source” and search for “All_Results”. This is the beginning tag of the XML we are after, copy that XML all the way to the ending </All_Results> and paste that into a new XML file named something like results.xml (leave the XML prelude if using VS to create the XML file).

So at this point you have the XML and a mockup of what you want your HTML to look like – so lets solve for the unknown, the XSLT. Now if you thought I was going to teach you XSL via a blog post I am sorry to disappoint but that topic will not scale to what I can do on this blog. There are tons of sources on the Internet and books which will do a much better job that I can ever do so I will redirect you to those at this point.

As for tools for creating XSLT I use Visual Studio 2010. I start by creating an empty project and importing my XML file into the project and then either create a new XSLT file or start from an existing file as I did in my previous post. What is really cool about using Visual Studio 2010 is that you get all the color coded XML and XSLT plus Itellisence. And if that were not enough you can also debug your XSLT using the XSLT Debugger (this was in VS 2008 too).

With your XSLT selected choose the XML menu item and “Start XSLT Debugging”. The first time you will be asked to supply a source XML file. VS 2010 will allow you to set breakpoints, execute, and step through the XSLT just as you can do with many of the other development languages VS supports. For example, take a look at the following screen shot, I am about to call into a template named FormatRating where I have set a breakpoint and VS has broken into. (click on the picture to make larger). As you can see I have setup 3 windows much like the pipeline concept I mentioned earlier: XML –> XSLT –> HTML. Notice how I can see the point within the XML along with the next line to be executed within the XSLT along with the current HTML buffer which has been flushed.

Using F5, F11, F10, and Shift-F11 I can execute, step into, step over, or step out of XSLT calls. In this case hitting F11 drops me into the FormatRating template where I can step over several variable initialization lines. From this next screenshot you can see how I added a few watch variables, the first two of which were parameters which were passed into my template and the last few are template local variables which I have manipulated some data into a desired format.

So just to wrap this up using Visual Studio 2010 with a source XML file and a solid idea of what you want to produce is a great way to create custom visualizations for many different SharePoint components which leverage XSLT.

Pimp Document Ratings within SharePoint Search Results

Tue, 05 Oct 2010 04:05:00 -1000

A good friend of mine Steve Peschka did a blog post during the SharePoint 2010 Beta which shows how to include a document’s rating within search results without writing any code. Well its been almost a year since that post and allot of things have changed. Since we RTM’ed SharePoint 2010 there are several key components to that blog that no longer work and this post is an attempt to update and hopefully add a little to the effort he started.

Turn on Ratings

To get started obviously we need Rating turned on in at least one list or library. For this example I will use a document library were I have uploaded a number of PowerPoint presentations and a few documents. To enable ratings I need to navigate to this document library and choosing “List settings” from the ribbon. From the Document Library settings page I choose “Rating settings” and I am presented with a super easy page to enable Ratings:

Once this setting is made a new column is added to my list and to the default view and at this point I can start rating content.

You may notice that once you start selecting a rating and then choose to refresh the page the rating might not “stick”. As you click on the stars and add a rating this is logged via an AJAX call into the social Database’s SocialRatings table. A timer job called the Social Data Maintenance Job which has the responsibility for walking through the SocialRatings table and calculating the average ratings and storing the results in the SocialRatings_Averages table. Its this table which is used to display the rating information which you see in the screenshot above and as a result any new rating will not be immediately visible on the web until the rating average has been calculated and move into this table.

Setting up Search

The next step is to create a Managed property in search that we can include in our search results. To have a Managed property you must first have a Crawled property so if you just turned on Rating for the first time and have yet to conduct a Full Crawl of search you need to do that now. If you are not sure navigate to your Crawled Properties by heading over to the search administration page within Central Administrator and choosing the Metadata properties link on the left navigation. Then choose the Crawled Properties link at the top and do a search for “rating”.

You should see two hits, one for ows_AverageRating and another for ows_RatingCount. If you don’t see these Crawled Properties kick off a Full Crawl in Search and wait for it to complete.

So lets add a Managed Property for each of the two Crawled Properties. First we will begin with the ows_AverageRating and create a Managed Property named '”Rating”. This value is of a decimal type and we need to choose the ows_AverageRating Crawled Property.

Next we will add a Managed Property named “RatingCount” which will be of type Integer and map it to the ows_RatingCount Crawled Property.

The last step here is to ensure both of our Crawled Properties are included in the index. We do this by selecting the “Include values for this property in the search index” checkbox on each Crawled Property.

After all of these changes have been made we need to do a Full Crawl of Search – this is super important; an incremental crawl will not suffice.

Customizing the XSLT

So while we are waiting on the Full Crawl to complete lets do some work to add the rating to the output of our search results. To do this we need to modify the XSLT of the Core Results WebPart on a search results page. Navigate to the search results page you plan on using and put the page into edit mode. Edit the properties of the Core Results WebPart by expanding the “Display Properties” section. We want to make 3 changes here:

Uncheck the “Use Location Visualization” option
Edit the Fetched Properties by adding the two new Managed properties to this list, for example: <Column Name="Rating"/><Column Name="RatingCount"/>
Using the XSL Editor button we will add our new customized XSLT (source included below).

So the first two changes are relatively trivial to pull off and get right, but its this third change which is the money configuration change you need to get right so I will spend a bit more time talking about this one.

This first step to customizing the CoreResults WebPart’s XSL is to save the default XSLT. Please do this, I know backing up or saving data when you may never need it is not sexy but it only takes maybe one minute of time and if you put it in a safe spot you never need to do it again. Sure you could just add another CoreResults Webpart and pull the default XSLT from that but that will take more time.

With the default XSL saved off we will make a copy of this file as the start of our customized XSLT. Word of warning here, XSL is not the most strait forward language to learn as you are using XML syntax to manipulate data. It takes a bit of getting use to but its not too hard to figure out especially when you have a large XSLT here to play around with and look for examples. The XSL we are going to add in this example is in two separate chunks – the first is the template and the second is the callout to the template itself. If you are a developer you will liken this to a method definition and a call to the method where the method definition is the template.

We begin editing the XSLT by adding the callout to our template. Within the “Result” template is where we want to insert this first bit of customization. Specifically we want to place it just after line 407 which should be a closing choose tag, eg: </xsl:choose> but before the first of the two </div> tags.

<!--Start Rating-->
<xsl:call-template name="FormatRating">
  <xsl:with-param name="theRating" select="rating" />
  <xsl:with-param name="theRatingCount" select="ratingcount" />
</xsl:call-template>
<!--End Rating-->

So with the template callout in place its time to add our template. We want to add this to the bottom of the XSLT just above the last tag which will be the stylesheet tag.

 <!--Start Rating-->
  <xsl:template name="FormatRating">
    <xsl:param name="theRating" />
    <xsl:param name="theRatingCount" />
    <xsl:if test="$theRating > 0">
      <div style="display: inline-block; padding-left: 8px;">
        <div>
          <xsl:variable name="tempTitle" select="concat($theRating, ' Stars (')"/>
          <xsl:variable name="tempTitle2" select="concat($theRatingCount, ' Ratings)')"/>
          <xsl:variable name="ratingTitle" select="concat($tempTitle, $tempTitle2)"/>
          
          <xsl:attribute name="title">
            <xsl:value-of select="$ratingTitle"/>
          </xsl:attribute>
          <xsl:choose>
            <xsl:when test="round($theRating) = 0 and $theRating > 0">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-368px 0px;width:16px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 1 and round($theRating) &lt;= $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-352px 0px;width:16px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 2 and round($theRating) > $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-352px 0px;width:32px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 2 and round($theRating) &lt;= $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-336px 0px;width:32px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 3 and round($theRating) > $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-336px 0px;width:48px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 3 and round($theRating) &lt;= $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-320px 0px;width:48px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 4 and round($theRating) > $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-320px 0px;width:62px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 4 and round($theRating) &lt;= $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-304px 0px;width:64px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 5 and round($theRating) > $theRating">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-304px 0px;width:80px;</xsl:attribute>
            </xsl:when>
            <xsl:when test="round($theRating) = 5">
              <xsl:attribute name="style">background-image:url(/_layouts/images/Ratings.png);cursor:default;height:16px;margin:0px;background-repeat:no-repeat;background-position:-160px 0px;width:80px;</xsl:attribute>
            </xsl:when>
          </xsl:choose>
        </div>
      </div>
    </xsl:if>
  </xsl:template>
  <!--End Rating-->

Once both edits are in place you can save the XSL and place into the Core Results WebPart.

So what is this XSL actually doing? Well the template (or method if you so choose) is named FormatRating and it takes two parameters. The first is the rating and the second is the rating count both of which are pulled from the XML sent to us as part of the search results from a query. The FormatRating template uses these two values to add an <div> into the resulting HTML. The background of the div is an image named Ratings.png which is part of the SharePoint 2010 installation. This image is a sprite and contains a few images within a single image file so the file is only downloaded once from the server but visually appears as many different images.

Each star is 16x16 and I use CSS styling to index into the image and show from 1 to 5 stars as well as half stars were necessary. All position and widths are divisible by 16 as I move the mask around to show different parts of the image based on the number of stars each item has been rated. I use the title property on the div to display the number of stars as well as the number of ratings.

The Results

As you can see from the screenshot below we now have rating data returned within the search results and we did all of this without writing any code (this is true if you don’t consider XSL code, otherwise I just lied to you).

UPS Complains After Installing the SharePoint 2010 August CU

Mon, 04 Oct 2010 03:15:00 -1000

The SharePoint 2010 August CU has a good number of User Profile Service (UPS) fixes and is worth installing. You may however notice a couple of fun error messages which start to appear to fill your Windows Application event logs, for example:

Log Name: Application
Source: Forefront Identity Manager
Date: Date
Event ID: 3
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: Server
Description:Microsoft.ResourceManagement.Service: Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManagerException: Forefront Identity Management Service does not support workflows of type 'Microsoft.ResourceManagement.Workflow.Activities.SequentialWorkflow, Microsoft.ResourceManagement, Version=4.0.2450.5, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHost(ResourceManagementWorkflowDefinition workflowDefinition)
at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.StartWorkflowInstance(Guid workflowInstanceIdentifier, KeyValuePair`2[] additionalParameters)

There was a binary mismatch in the Aug CU which caused this issue and the good news the error messages, while pervasive can be safely ignored. You may have the Aug CU and not have these error messages even though you are using UPS and the reason is most likely because you created the UPS after installing the Aug CU. One workaround for this issue, until a new fix is released (most likely the Oct CU??), is to re-provision the UPS however I would NOT recommend this approach for a good number of reasons including but not limited to potential data loss. If you want a reminder of some of those reasons search around this blog and many others for issues with the UPS to get a refresher.

Microsoft PSS has released a KB article detailing the issue.

Why I don’t Dev SharePoint 2010 on Windows 7

Mon, 06 Sep 2010 03:17:00 -1000

Recently some of the MCM SharePoint masters were kicking around different development environments and setups for SharePoint 2010. This was an interesting conversation because of SharePoint 2010’s support for Windows 7 and Windows Vista as an OS for development and test purposes. For more information on how to set this up check out this MSDN article. In some circles SharePoint development on Windows 7 has become the platform of choice due to Microsoft’s lack of an offering for x64 virtualization on Windows 7. For me personally however running Windows 7 as a SharePoint development environment was not ever that compelling for a number of reasons:

My main computer is a laptop and I travel quite a bit. So I am one of those mobile developer types and I don’t want OWSTimer, SQL Server, IIS, etc running and consuming memory while I am in the airport on battery trying to determine why my flight is late, again. I am too lazy and forgetful to run scripts to toggle services off and one each time I go from battery to full power and back.>
I work with many different customers whom all seem to run different SharePoint SKUs (Foundation, Standard, Enterprise) as well as different patch levels. I like the ability to test a solution on the same software SKU and patch level which my customer is running. I also like the ability to roll back a patch if necessary should something go bad or if I have a private fix which I need to test. Running Windows 7 on my laptop and installing SharePoint 2010 would lock me into one SKU and patching would be a forward only excise without any ability to rollback.
There are some cases where running on Windows 7 just will not work, for example, User Profile Service (UPS) and Office Web Applications typically do not work without some additional effort.
When I hit a strange error is it my Windows 7 configuration, a stupid dev anomaly, or is it really something I need to track down? I am sure the testing of SharePoint on Windows 7 was not as rigorous as what was performed on Windows Server 2008/R2. I have been around the product servicing game long enough to know that if I find a “SharePoint on Windows 7” bug I will have a heck of a time justifying the cost and priority to have that bug fixed unless it impacts more than just myself and a small hand of folks. That said, I currently do not know of a single bug which reproduces on Windows 7 but not on Windows 2008/R2 so don’t ask for a list.
I like being able to hit the reset button on a dev environment to cleanup stuff – I may attach debuggers, turn on logging, make registry settings, etc. My mind slips quite often and it’s just easier to reset to a snapshot rather than trying to back out everything I may have done to the system. If you have ever set a custom route, IP address, DNS setting, host settings, etc. to test something and days later forget and call down to the hotel front desk to complain about their crappy Internet you know what I am talking about. Windows 7 new backup application is really nice and I use it quite often however that is a disaster recovery solution and not a solution I would try to apply to this problem.
Microsoft, like so many other companies, push down domain policies such as firewall, screensaver, IPSec, Bitlocker, etc and reserve the right to add new policies at any point. I am confident that when IT departments are considering a new domain policy they are not thinking about those folks running SharePoint on Windows 7. Bitlocker, IPSec, Firewall, and strong password policies in a development environment only get in the way which, for me, reduces my already pitiful productivity and therefore increases costs (fortunately I come cheap so “costs” are relative).
I like running debug logging, performance logs, Netmon sniffs, etc. on a separate machine so I don’t have to filter out all that noise which comes from Office communicator, Outlook and all those other applications I have to run to do my job.

So while Windows 7 as a SharePoint development platform is not for me however I would encourage everyone whom is considering a SharePoint development platform take a look at Windows 7 as my reasons may not be your reasons. I have heard from quite a few folks whom are happy with SharePoint on Windows 7 so you should test and investigate for yourself – besides basing such an important decision on a blog post is never recommended.

SharePoint 2010 Cumulative Update #1 Released

Thu, 22 Jul 2010 07:02:00 -1000

Well it’s the middle of July so you know what time it is, its time for the SharePoint 2010 June CU.

Here are a few links…

Title	KB	Download
Foundation Server	KB2028568	Download
Search Server 2010	KB983319	Download
SharePoint Server 2010	KB2281364	Download
SharePoint 2010 Filter Pack	KB2124512	Download

Must Have: Visual Studio 2010 Power Tools

Tue, 13 Jul 2010 06:16:00 -1000

So for those that have not already done so installing the Visual Studio 2010 Power Tools is well worth your time, especially if you do any SharePoint development. Today I created a console application to test SharePoint 2010. One of the more annoying tasks when creating such an application is adding all the necessary references to the project -- mainly because this is very taxing for my brain to try to remember the various assemblies. One of my favorite features of the VS 2010 Power tools is the new Add Reference dialog (see below).

This screenshot is the new Add Reference Dialog and at the time I captured this view I had just done a search for “sharepoint” and without hitting a single button was returned this list of assemblies. There was not a long delay in loading the assemblies and I don’t have to contend with the different naming conventions used by the various PGs within MSFT.

“Enjoy the little things” — Zombieland

VS2010 Power Tools can be downloaded from: http://visualstudiogallery.msdn.microsoft.com/en-us/d0d33361-18e2-46c0-8ff2-4adea1e34fef

Get Your SharePoint 2010 Learn On

Wed, 12 May 2010 07:18:00 -1000

We just released IT Professional and Developer training which includes PowerPoint presentations and videos going over the presentations along with demos to break things up a bit.

The content is broken into two areas, IT Pro and Developer, and both contain hours of content all based on RTM. The content assumes you are familiar with SharePoint and is geared more as a “what’s new, What’s improved, and What’s available”. So rather than explaining what a Feature is all about you get just what you want without having to go through all that stuff you are probably already familiar with.

Where is it?

SharePoint 2010 Advanced Developer Training

SharePoint Server 2010 Advance IT Professional Training

Happy training – let us know how it goes!

Sasquatch Out!

Thu, 06 May 2010 07:54:00 -1000

In my previous post I documented a bug in SharePoint where a memory leak can greatly impact that amount of memory you SharePoint WFE will use during the normal processing of web request. The SharePoint PG just recently released the 2010 April CU for WSS 3.0 and MOSS 2007. It requires you have Service Pack 2 before installing these cumulative updates but I can tell you that is something you want to invest in.

WSS 3.0
Main KB: http://support.microsoft.com/kb/981043
Download Link: Download April 2010 Server-Package for WSS (981043)

MOSS 2007
Main KB: http://support.microsoft.com/kb/982741
Download Link: Download April 2010 Server-Package for MOSS 2007 (981042)

If you dig into the description of what has been fixed in the CUs you will come across this KB which has this little blurb regarding the Sasquatch.

A memory leak may cause the application pool to recycle the AppDomain because many instances of the SPHttpApplication reach memory limits.

I cannot stress enough how much everyone running SharePoint 2007 needs to apply this fix.

Searching For An Error?

Tue, 04 May 2010 06:40:00 -1000

So lets say you create a new site based on the Team template and you would like to create a search center site under that same site collection. So you navigate to Site Actions – New Site and maybe you choose the Enterprise Search Center template and provide a name for your new site and hit “Create”. Well you may be presented with an error like this one…The dreaded unexpected error has occurred error, yuk (failure #1).

Well no worries right, you have the ULS log to query for a hint. While the dialog here provides a correlation ID you can look for in the ULS logs you cannot copy it from this dialog (failure #2) but you can still query the ULS log for entries that match this ID and you will hit upon an entry which a nice looking stack – once you factor the stack out of the ULS entry it will look like the following.

Detected use of SPRequest for previously closed SPWeb object.  Please close SPWeb objects when you are done with all objects obtained from them, but not before.  
Stack trace:    
at Microsoft.SharePoint.WebControls.ScriptLink.SharePointClientJs_Register(Page page)     
at Microsoft.SharePoint.WebControls.ScriptLink.InitJs_Register(Page page)     
at Microsoft.SharePoint.WebControls.ScriptLink.RegisterForControl(Control ctrl, Page page, String name, Boolean localizable, Boolean defer, Boolean loadAfterUI, String language)     
at Microsoft.SharePoint.WebControls.ScriptLink.Register(Page page, String name, Boolean localizable, Boolean defer, Boolean loadAfterUI, String language, String uiVersion)     
at Microsoft.SharePoint.WebControls.ScriptLink.RegisterOnDemand(Page page, String strKey, String strFile, Boolean localizable)     
at Microsoft.SharePoint.WebControls.ScriptLink.RegisterCore(Page page, Boolean defer)     
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.RegisterOWSScript(Page page, SPWeb web)     
at Microsoft.SharePoint.WebPartPages.WebPartPage.FormOnLoad(Object sender, EventArgs e)     
at System.Web.UI.Control.OnLoad(EventArgs e)     
at System.Web.UI.Control.LoadRecursive()     
at System.Web.UI.Control.LoadRecursive()     
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     
at System.Web.UI.Page.ProcessRequest()     at System.Web.UI.Page.ProcessRequest(HttpContext context)     
at ASP._layouts_addgallery_aspx.ProcessRequest(HttpContext context)     
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)     
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)     
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)     
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)     
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

So this does not really give us anything to go on here but it would appear there is a bug where someone is probably disposing of an SPContext.Current object – still nothing we can do about that.

So lets attach WinDbg to the correct w3wp process and retry the operation. Once reproduced lets look at what was spit to the debugger….

<<snip>>
(14f4.11c0): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
CLR exception type: System.InvalidOperationException
    "The SharePoint Server Publishing Infrastructure feature must be activated at the site collection level before the Publishing feature can be activated."
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.ec0): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.11c0): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)
(14f4.11e8): CLR exception - code e0434f4d (first chance)

OK now I have something to go look at. Sure enough on my Team site I did not have the Site Collection Feature SharePoint Server Publishing Infrastructure enabled. Once I enabled this feature I retried the operation and sure enough the problem was resolved. So failure #3 was assuming the ULS was giving me the best error available. I suspect the error being reported was due to a bad code path which we took once the source error was encountered.

Yet Another Possible User Profile Sync Error You May See

Tue, 04 May 2010 04:31:00 -1000

I Have had the great honor of hitting a good number of errors while setting up the User Profile Sync service. I documented one error here but when setting up another environment today I hit yet another. The environment in question was a two server setup with one server being the DC and the other SharePoint + SQL Server. I use the farm account CONTOSO\FarmAccount. I setup all Service Applications using the PowerShell script found here and started the services I wanted to run. When I got to starting the User Profile Synchronization Service I would eventually see the following error in the event log.

With only this error to go on I decided to attach a debugger. Now that was probably heavy handed and ProcMon would have have probably given me the answer here too but WinDbg is much more sexy, yea! Soon after I started the service with windbg attached to the owstimer service I found a security error trying to access the registry. Poking around a bit in the debugger I was able to determine the CONTOSO\FarmAccount was attempting to gain access to HKLM\SYSTEM\CurrentControlSet\Services\FIMSynchronizationService\Parameters and was failing. I decided to shotgun this one and grant my CONTOSO\FarmAccount local admin, restarted the timer service, and retried the operation – it worked. Thinking back about what may have happened I opened the key within the registry editor…now what is strange is I noticed the CONTOSO\FarmAccount has permissions to that key – I wish I had looked before I fixed the problem however I suspect the permissions might have been set after the provision completed or at least got past the point where I failed the first time.

If anyone hits this error, before you do what I have done here I would check the permissions on HKLM\SYSTEM\CurrentControlSet\Services\FIMSynchronizationService\Parameters and see if your Farm Account is present.

Give your Application Pool Accounts A Profile

Mon, 03 May 2010 09:48:00 -1000

While running SharePoint 2010 I started to notice the following error messages in my Application event log each time I restarted the IIS Worker process and made a request.

Log Name:      Application
Source:        Microsoft-Windows-User Profiles Service
Date:          5/3/2010 10:05:07 AM
Event ID:      1511
Task Category: None
Level:         Error
Keywords:
User:          CONTOSO\AppPoolAccount
Computer:      SP2010Dev.contoso.com
Description:
Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.

For the setup of my SharePoint farm I use 3 accounts, Farm (CONTOSO\FarmAccount), SA Service Account (CONTOSO\SPServiceAccount), and an Application Pool account (CONTOSO\AppPoolAccount). Looking at my c:\Users directory I see that I have a FarmAccount profile so if you are running your Application Pools as this account then you probably are not seeing this error message. But if you are using a seperate account to run your Application Pools you are probably seeing this error in your event logs.

There is obviously something running with the w3wp process which needs to load the user’s profile, which in this case is my CONTOSO\AppPoolAccount. After creating a profile for my AppPoolAccount this error disappeared and I confirmed with ProcMon that it was loaded and being used by w3wp.

There are obviously several ways to create a profile for a user, for example you might choose to login to the server as that user however I chose a different path which seemed faster.

Run IISReset and ensure the AppPool account is not running any processes and its temporary profile is not loaded.
If your AppPool account does not have login locally permission you will need to grant it for this process, I just use the cmd line: net localgroup administrators CONTOSO\AppPoolAccount /add when complete I ran the same command but with a /delete
From the command line run: runas /u:CONTOSO\AppPoolAccount /profile cmd and then type the password when prompted. At the point the cmd prompt opens you should have a new profile for this user at c:\users\AppPoolAccount, if not then within your cmd prompt that you just started run echo %userprofile% if this does not work then the real profile was not loaded and you are still probably using the temporary profile. Try deleting the temporary profiles out of c:\users, these are probably named c:\users\Temp.[domainname].00x, example c:\users\temp\contoso.002 After you delete these try this process again.
Once you have a profile you may want to remove any logon locally rights you may have granted your AppPool account.

Help Wanted: Taxonomy Picker

Mon, 03 May 2010 04:18:00 -1000

On SharePoint 2010 you may run into the following error (hell that is a damn lie, you will hit this error eventually if running RTM).

Load control template file /_controltemplates/TaxonomyPicker.ascx failed: Could not load type 'Microsoft.SharePoint.Portal.WebControls.TaxonomyPicker' from assembly 'Microsoft.SharePoint.Portal, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'.

It looks like this in your Application Event Log:

UPDATE:: After doing this fix I received this error again. I checked the Microsoft.SharePoint.Portal assembly and did not find the TaxonomyPicker class!!! So to fix this you may want to rename TaxonomyPicker.ascx to TaxonomyPicker.ascx_broken so we do not try to recompile it each time the AppPool is started. The user control is never used within SharePoint and therefore serves no real purpose.

If you assumed you configured something incorrectly; don’t, its a bug in SharePoint – the good news is you can actually fix this one yourself.

Navigate to /14/TEMPLATS/ControlTemplates/TaxonomyPicker.ascx user control
Open the user control in a text editor and locate the first line
Find the character string , and replace with a comma ‘,’ (without quotes).
Save the user control and you have completed fixing this bug

Stick With What Got You Here

Sat, 01 May 2010 03:44:00 -1000

I think its safe to assume most SharePoint developers which have been working with the product for any length of time have seen code similar to the code below below. Its a common problem, disposable objects which are not disposed. There has been a ton written on this problem and Roger Lamb wrote a great tool called SPDisposeCheck which you run against your assemblies to determine if they may be failing to call Dispose.

public partial class VisualWebPart1UserControl : UserControl
{
  protected void Page_Load(object sender, EventArgs e)
  {
    SPSite site = new SPSite("http://intranet.contoso.com");
    foreach (SPWeb web in site.AllWebs)
    {
      Response.Write(web.Name);
    }
  }
}

Visual Studio 2010 includes a new Code Analysis rule which can check for these problems too. Now I am no SKU king so I cannot tell you what SKUs come with Code Analysis and which do not. I do know three things about this topic, (1) Not all Visual Studio 2008 SKUs included this feature, (2) I am guilty of being one of the many Microsoft minions which when we install Microsoft products look for versions with names like “Enterprise”, “Datacenter”, “Ultimate”, “Uber”, etc. (3) I am running Visual Studio 2010 Ultimate and it has Code Analysis built-in.

You can enable Code Analysis from the Project Properties dialog and choosing the “Code Analysis” tab at the bottom and then clicking on the “Enable Code Analysis…” checkbox which I have highlighted here.

For the code I have include above here is what Visual Studio Reports…

At this point a pain of disappointment should be felt; Visual Studio 2010 only found one of the two Dispose issues, that is a 50% success rate if you are that type of person; if you are the other that is a 50% fail rate – pick your side but it is unacceptable for such as small and trivial test.

Now how does SPDisposeCheck fair? SPDisposeCheck found both issues!

Conclusion
Although Visual Studio 2010 has many great code analysis rules, and I use them frequently I personally would continue to use SPDisposeCheck as part of my development toolset.

Don’t Claim to Be Someone You are Not

Fri, 30 Apr 2010 08:19:00 -1000

In SharePoint 2010 you have the option of setting your Web Application to use Claims based security or Classic security which is the same as Windows security, aka like we did it in SharePoint 2007. While playing around with an anonymous site which was hosted within a Web App which was configured for claims I came across something interesting that I wanted to make sure you all were aware of. Although your Web Application is configured for claims your OS still uses Windows authentication, this means your AppPool and your IIS anonymous user are not using Claims.

To make the point check out the bit of code below and note right before the local variables the values which will be stored for each once this code is executed within an anonymous site within a claims based web application.

    
    public partial class DaWebPart : UserControl
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            /*NULL*/
            string claimAnon = System.Threading.Thread.CurrentPrincipal.Identity.Name;

            /*NT AUTHORITY\IUSER*/
            string winAnon = System.Security.Principal.WindowsIdentity.GetCurrent().Name
            
            /*NULL*/
            string runWithClaim;

            /*CONTOSO\AppPoolAccount*/
            string runWithWin; 

            SPSecurity.RunWithElevatedPrivileges(delegate
            {
                runWithClaim = System.Threading.Thread.CurrentPrincipal.Identity.Name;
                runWithWin = System.Security.Principal.WindowsIdentity.GetCurrent().Name;
            });
        }
    }

This scenario does not change much when not running anonymous other than you will have real user names when you access your Claim Identities via System.Threading.Thread.CurrentPrincipal.Identity.

So when we make a call to RunWithElevatedPrivileges() its doing the right thing for us however if you only look at the Thread’s CurrentPrincipal.Identity you will be looking at the ClaimIdentity but you just told SharePoint to change out your WindowsIdentity. An important distinction if you take a dependency on this functionality when moving your code from 2007 to 2010 with claims.

A Session State By Any Other Name

Fri, 30 Apr 2010 04:01:00 -1000

Recently Mark Arend and I were helping out a co-worker setup Session State on SharePoint 2010 and he pointed out something very important; while I was providing the means to enable the “State Service” I was not providing the method to turn on ASP.Net session state. At first I was not clear on the distinction however once I read his email for the 3rd time after my second cup of coffee I finally got it..while I was using New-SPStateServiceDatabase to setup the State Service DB he was suggesting to use Enable-SPSessionStateService. Yea it looks pretty clear now that these are two very different commands however in my stupor after working on User Profile Sync issues over the past couple of days these were equal in my feeble mind.

So what’s the difference? The State Service is used by InfoPath Forms Services (IPFS), the Chart Web Part, and when the user is not running Silverlight Visio Services will use this too. The ASP.Net session state is used within aspx pages, controls, Web Parts, etc to store user/session scoped state such as when you make a call to Page.Session.Add(“Key”, new object()); As developers, we have access to the Asp.Net Session state only while the State Service is reserved for the Office Server applications mentioned previously.

How to Enable SharePoint’s State Service

Out of the box (OOTB) you may find you receive an error stating Session State is not enabled. This error is referring to the SharePoint State Service which needs to be enabled. If you use the Configuration Wizard you will see this choice present along with numerous other services however if you choose to not enable it through the Configuration Wizard you will need to use PowerShell. For some reason this service does not appear as on option on the “New” ribbon button within the Service Application. In my previous blog post I provided a PowerShell script which setup the State Service. While there are quite a few verbs to this commandlet you only really need the “New” to install/setup this type of session state.

Here is a sample command which installs the State Service:

####
##Install new State Service
####

New-SPStateServiceDatabase -Name "StateSvcDB" -DatabaseServer "db1" | New-SPStateServiceApplication -Name "State Svc" | New-SPStateServiceApplicationProxy -Name "State Svc Proxy" -DefaultProxyGroup

How to Enable ASP.Net Session State

The only way to install ASP.Net Session State is through PowerShell using *-SPSessionStateService. [Enable | Disable | Get | Set]-SPSessionStateService

Using this example I was able to install the ASP.Net Session State on my development farm. Note you must pass a database name which does not include any spaces. If you happen to do this you will receive an error and you may notice additional SQL Server Agent Jobs (which I will discuss later) which are created each time you try to pass off a DB name with a space (jeez you think I actually did this or what).

####
## Enable session state
####


Enable-SPSessionStateService -DatabaseName "ASPNet_Session_State" -DatabaseServer "db1" -SessionTimeout 30

Once enabled each web.config file is modified by adding the System.Web.SessionState.SessionStateModule to the list of Http modules. In addition a new sessionState element is added similar to the following:

<!--

web.config settings

-->
<sessionState mode="SQLServer" timeout="30" allowCustomSqlDatabase="true" sqlConnectionString="Data Source=db1;Initial Catalog=ASPNet_Session_State;Integrated Security=True;Enlist=False;Connect Timeout=15" />

Notice I said “each” web.config file is modified, that is, all root web.config files for all Web Applications are modified, for example see the following image. When new Web Applications are added later they too will receive this web.config modification when provisioned.

Now this does not mean ASP.Net Session state is available on any web application out the gate. An administrator would have to enable session state at each web application by locating the “pages” element such as this: <pages enableSessionState="false" and changing the enableSessionState to true. After making this change Session state is now available to be used.

More Information
When using Visual Studio 2010 to create SharePoint projects you can create aspx pages using designers such as the Application Page designer. These designers create pages with a <%@ Page tag which does not supply an EnableSessionState attribute, the default for this attribute is “true” so for almost all cases you have just enabled session state for all your pages by making the change above. SharePoint 2010 uses the OOTB ASP.Net Session State provider and Session State HttpModule. This means State is binary de-serialized once at the beginning of a page request and serialized back once the request is complete. This happens on any page where 1)session state has been enabled on the farm, 2) enableSessionState is true on the web application, 3) there is a valid session state cookie being sent from the client, 4) the Page does not include an EnableSessionState attribute or the attribute is set to true.

If you decide to use session state it is prudent to ensure a) its enabled (Page.Session != null) and b) you explicitly set enableSessionState to “false” where you don’t need session state or “ReadOnly” where you only need a read only copy of the session state were you do not plan to write back to (note you can do Session.Add(..) or Session[key]=”” within a page with ReadOnly set, the changes just will not be persisted). When “true” or not present two round trips are made to the Session State DB for each page request. Setting this value to “false” eliminates these round trips while setting it to “ReadOnly” means we only fetch the session state from the DB on page request and throw it away when the request completes meaning you can cut the number of round trips in half.

Scale is important here as there is only one Session State DB per farm to be used for all Web Applications, so investigate other means of storing state before enabling this feature. Session State is track via a cookie sent back to the browser when session state is stored for the client, this occurs for each web application which uses session state. So if you have my.contoso.com and intranet.contoso.com which both happen to use session state, both are on the same farm, they will use different session state cookies and different session state objects stored in the same session DB. Since session is stored and shared within a Web Application operations performed on the session state such as Session.Clear(), Session.Abandon(), etc can potentially impact other applications, pages, or sites which use session state. Keep this in mind if you have developers complaining about losing session state – its probably not SQL or ASP.net’s fault, its probably someone trying to be a good citizen and clearing out the session state.

Session Expiry is handled by a SQL Server Agent Job which is installed when you enable the service. This job runs once a minute cleaning up any expired sessions. It’s important your SQL Server Agent service is running. This was not always the case for MOSS 2007 and an important change here; otherwise your ASP.Net session state will not be cleaned up meaning your database will grow, and grow, and…you get it.

Automated Install of User Profile Sync can be a “Challenge”

Thu, 29 Apr 2010 05:01:00 -1000

Over the last two days I have had a hard time setting up User Profile Sync on SharePoint 2010. I chose not to run the wizard or use the UI to install the service but instead chose to use this PowerShell script to setup the Service Application’s and Proxies and then start the User Profile sync service. Now this is where things went bad…

Let me take you through the steps I went through so hopefully you can avoid these problems.

I ran the Prereq installer and the SharePoint 2010 setup to lay down the binaries.
Took a snapshot (yea I do everything on a VM and after you read this blog post it will become very clear why Windows 7 installs just will not cut it)
I ran PSConfig
Took a snapshot
Ran this PowerShell script.
Attempted to start the User Profile Synchronization Service, and I waited…
While I waited I checked the Application event logs and found numerous warning messages source from the “ILM Web Service Configuration” with descriptions similar to “ILM Database could not be created: Error sent to Windows Event Log running:” and including various details such as: “IF EXISTS (SELECT * FROM sys.xml_schema_collections c, sys.schemas s WHERE c.schema_id = s.schema_id AND (quotename(s.name) + '.' + quotename(c.name)) = N'[dbo].[StringSchemaCollection]') DROP XML SCHEMA COLLECTION dbo.StringSchemaCollection: Specified collection 'StringSchemaCollection' cannot be dropped because it is used by object 'CONTOSO\FarmAccount.GetObjectCurrent'.”
What was really strange about this error was the object name CONTOSO\FarmAccount.GetObjectCurrent so when I navigated into my SyncDB I found something very odd, notice the stored procedure names in the image below. Now for the object in question here it was actually a view but the same problem none the less.
I am using the CONTOSO\FarmAccount as my Farm account which is also the account I used to start the Profile Sync service. Using Process Monitor I discovered the SQL scripts which generate this DB are coming from %programFiles%\Microsoft Office Servers\14.0\Sql\ When I search through them I found the objects_views.sql which is what created my offending view(s) from the error message I provided previously.

Something very important to note about these scripts, in the CREATE VIEW statements the name of the view is given without specifying the schema. This means SQL server will use the default schema for the user running this command.
So what is the default schema for CONTOSO\FarmAccount?
At this point I am pretty well hosed so I rollback to the snapshot I took just before running the PowerShell script (#4 in this list) and I reran the Script again.
Now at this point I am going to do something that will cast you in the realm of '”The Unsupported” so under no circumstance do I recommend you do what I am about to do on a production farm or a farm where you may want to receive any kind of support from Microsoft.
So dude, why are you showing us this at all then? I am showing this merely as education. This is clearly a bug and something I hope to see fixed in the next CU.

So at this point I changed the default schema for CONTOSO\FarmAccount to “dbo”.
I then started the Profile Sync service again and waited again…
This time the DB provisioned perfectly with no errors on the Profile Sync server was “running” – how sad that I jumped out of my chair when I noticed the service went from the dreaded “starting” to “running”; but it happened anyhow.
I then reset the schema back to the default setting of CONTOSO\FarmAccount (note this is specific to my installation and yours may be different).
Ran IISReset (for SharePoint you cannot do this too many times)
Opend the User Profile SA, added a connection to my local AD and synced successfully!

Now is this issue specific to my single-server setup, I don’t think so but I will wait to hear from you guys on that part. What I do think is a contributor is using PowerShell to setup the SAs. So does this mean my PowerShell is broken? Well maybe, though I don’t think its too bad I will point out at the time of this post there is little to no PowerShell documentation on setting up each Service Application that I could find Binging around the Inter-web, but again I will wait to hear your feedback.

I have raised this issue to the product team and waiting to hear how we are going to handle this one going forward. I will keep you guys posted.

Keep it Short When Dealing With SharePoint’s PSConfig

Wed, 28 Apr 2010 04:47:00 -1000

Ran into an issue today when setting up my SharePoint 2010 development VM that I suspect a few others may hit as well. I found that no matter what I did I could not get the User Profile Synchronization service to provision. My dev machine is a single-server everything, that is, its a DC, DNS, SQL and SharePoint.After struggling with this for a good while I booted up another VM which was based on RC and discovered that when I navigated to the Services on Server page it showed the name as “sp2010dev” while on my broken box it was “sp2010dev.contoso.com”.

Thinking back a bit -- I did a full farm install of SharePoint and when I ran PSConfig I supplied to DNS domain name as my SQL server to host the Configuration DB which also was the location machine name “sp2010dev.contoso.com”. This meant my machine was added to the farm as sp2010dev.contoso.com. Fortunately I backed back out to a snapshot I had before I ran PSConfig and I took another run at this problem but this time I supplied to NetBIOS name of “sp2010dev”. This time when I went into the Services on Server page my server showed up as “sp2010dev”. So do you think this fixed my issue? Duh, I am blogging about it aren’t I? Within 3 minutes of starting the User Profile Synchronization Service it went from “Starting” to “Started” and the service was provisioned!

My Repro
1. Run PSConfig on a machine that is also your SQL server.
2. Within PSCOnfig supply the DNS name for your SQL server (which is also the local machine, eg. sp2010dev.contoso.com )
3. Note in services on server the name of the machine is the DNS server name eg. sp2010dev.contoso.com
4. At this point you will never be able to provision/start the User Profile Sync service.

Fix
Ensure you always supply the short name (sp2010dev) into PSConfig and not the DNS machine name!

The Wizard Likes His GUIDs

Mon, 26 Apr 2010 08:34:00 -1000

SharePoint 2010 includes a new Configuration Wizard which guides you configuring the SharePoint Service Applications. One of the downsides of running this wizard however is that the Database Names generated by the wizard contain a GUID in the name, for example “Performance Point {guid}”. If you are like me you are probably not being a big fan of naming something with a GUID when that object is typically read by humans. Each SA can be configured individually via the UI however a couple cannot, e.g. the State Service SA. I created the script below which will configure each SA (minus the Application Registry and Lotus Notes SAs) into a single Service Application, SQL server, and will start the services. While its probably a nice script to have for a single server development or test machine it will likely need to be changed for larger “real-world” deployments. I include it here however as an example to help you get started or if you have a dev box where you want to use GUID-less DB names. Please note that I did not take the time to configure each SA with the exact defaults as the Wizard – while totally possible I have a few other items on my plate but did want to share what I had as of now…

################################################################################
## This script replicates most of the functionality found in the Config Wizard##
################################################################################


Add-PSSnapin Microsoft.SharePoint.PowerShell -erroraction SilentlyContinue

## Settings you may want to change ##
$databaseServerName = "sp2010dev.contoso.com"
$searchServerName = $databaseServerName
$saAppPoolName = "SharePoint Web Services"
$appPoolUserName = "CONTOSO\SAAppPoolAccount"


## Service Application Service Names ##
$accesssSAName = "Access Services"
$bcsSAName = "Business Data Connectivity Service"
$excelSAName = "Excel Services Application"
$metadataSAName = "Managed Metadata Web Service"
$performancePointSAName = "PerformancePoint Service"
$searchSAName = "SharePoint Server Search"
$stateSAName = "State Service"
$secureStoreSAName = "Secure Store Service"
$usageSAName = "Usage and Health Data Collection Service"
$userProfileSAName = "User Profile Synchronization Service"
$visioSAName = "Visio Graphics Service"
$WebAnalyticsSAName = "Web Analytics Service"
$WordAutomationSAName = "Word Automation Services"



$saAppPool = Get-SPServiceApplicationPool -Identity $saAppPoolName -EA 0
if($saAppPool -eq $null)
{
  Write-Host "Creating Service Application Pool..."

  $appPoolAccount = Get-SPManagedAccount -Identity $appPoolUserName -EA 0
  if($appPoolAccount -eq $null)
  {
      Write-Host "Please supply the password for the Service Account..."
      $appPoolCred = Get-Credential $appPoolUserName
      $appPoolAccount = New-SPManagedAccount -Credential $appPoolCred -EA 0
  }

  $appPoolAccount = Get-SPManagedAccount -Identity $appPoolUserName -EA 0

  if($appPoolAccount -eq $null)
  {
    Write-Host "Cannot create or find the managed account $appPoolUserName, please ensure the account exists."
    Exit -1
  }

  New-SPServiceApplicationPool -Name $saAppPoolName -Account $appPoolAccount -EA 0 > $null
	
}




Write-Host "Creating Usage Service and Proxy..."
$serviceInstance = Get-SPUsageService
New-SPUsageApplication -Name $usageSAName -DatabaseServer $databaseServerName -DatabaseName "UsageDB" -UsageService $serviceInstance > $null




Write-Host "Creating Access Services and Proxy..."
New-SPAccessServiceApplication -Name $accesssSAName -ApplicationPool $saAppPoolName > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Access Database Service"} | Start-SPServiceInstance > $null




Write-Host "Creating BCS Service and Proxy..."
New-SPBusinessDataCatalogServiceApplication -Name $bcsSAName -ApplicationPool $saAppPoolName -DatabaseServer $databaseServerName -DatabaseName "BusinessDataCatalogDB" > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Business Data Connectivity Service"} | Start-SPServiceInstance > $null




Write-Host "Creating Excel Service..."
New-SPExcelServiceApplication -name $excelSAName –ApplicationPool $saAppPoolName > $null
Set-SPExcelFileLocation -Identity "http://" -ExcelServiceApplication $excelSAName -ExternalDataAllowed 2 -WorkbookSizeMax 10 -WarnOnDataRefresh:$true 
Get-SPServiceInstance | where-object {$_.TypeName -eq "Excel Calculation Services"} | Start-SPServiceInstance > $null




Write-Host "Creating Metadata Service and Proxy..."
New-SPMetadataServiceApplication -Name $metadataSAName -ApplicationPool $saAppPoolName -DatabaseServer $databaseServerName -DatabaseName "MetadataDB" > $null
New-SPMetadataServiceApplicationProxy -Name "$metadataSAName Proxy" -DefaultProxyGroup -ServiceApplication $metadataSAName > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Managed Metadata Web Service"} | Start-SPServiceInstance > $null




Write-Host "Creating Performance Point Service and Proxy..."
New-SPPerformancePointServiceApplication -Name $performancePointSAName -ApplicationPool $saAppPoolName > $null
New-SPPerformancePointServiceApplicationProxy -Default -Name "$performancePointSAName Proxy" -ServiceApplication $performancePointSAName > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "PerformancePoint Service"} | Start-SPServiceInstance > $null


##START SEARCH

Write-Host "Creating Search Service and Proxy..."
Write-Host "  Starting Services..."
Start-SPEnterpriseSearchServiceInstance $searchServerName
Start-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance $searchServerName

Write-Host "  Creating Search Application..."
$searchApp = New-SPEnterpriseSearchServiceApplication -Name $searchSAName -ApplicationPool $saAppPoolName -DatabaseServer $databaseServerName -DatabaseName "SearchDB"
$searchInstance = Get-SPEnterpriseSearchServiceInstance $searchServerName

Write-Host "  Creating Administration Component..."
$searchApp | Get-SPEnterpriseSearchAdministrationComponent | Set-SPEnterpriseSearchAdministrationComponent -SearchServiceInstance $searchInstance


##Crawl
Write-Host "  Creating Crawl Component..."
$InitialCrawlTopology = $searchApp | Get-SPEnterpriseSearchCrawlTopology -Active
$CrawlTopology = $searchApp | New-SPEnterpriseSearchCrawlTopology
$CrawlDatabase = ([array]($searchApp | Get-SPEnterpriseSearchCrawlDatabase))[0]
$CrawlComponent = New-SPEnterpriseSearchCrawlComponent -CrawlTopology $CrawlTopology -CrawlDatabase $CrawlDatabase -SearchServiceInstance $searchInstance
$CrawlTopology | Set-SPEnterpriseSearchCrawlTopology -Active

Write-Host -ForegroundColor white "  Waiting for the old crawl topology to become inactive" -NoNewline
do {write-host -NoNewline .;Start-Sleep 6;} while ($InitialCrawlTopology.State -ne "Inactive")
$InitialCrawlTopology | Remove-SPEnterpriseSearchCrawlTopology -Confirm:$false
Write-Host



##Query
Write-Host "  Creating Query Component..."
$InitialQueryTopology = $searchApp | Get-SPEnterpriseSearchQueryTopology -Active
$QueryTopology = $searchApp | New-SPEnterpriseSearchQueryTopology -Partitions 1 
$IndexPartition= (Get-SPEnterpriseSearchIndexPartition -QueryTopology $QueryTopology)
$QueryComponent = New-SPEnterpriseSearchQuerycomponent -QueryTopology $QueryTopology -IndexPartition $IndexPartition -SearchServiceInstance $searchInstance
$PropertyDatabase = ([array]($searchApp | Get-SPEnterpriseSearchPropertyDatabase))[0] 
$IndexPartition | Set-SPEnterpriseSearchIndexPartition -PropertyDatabase $PropertyDatabase
$QueryTopology | Set-SPEnterpriseSearchQueryTopology -Active


Write-Host "  Creating Proxy..."
$searchAppProxy = New-SPEnterpriseSearchServiceApplicationProxy -Name "$searchSAName Proxy" -SearchApplication $searchSAName > $null


#####END SEARCH


Write-Host "Creating State Service and Proxy..."
New-SPStateServiceDatabase -Name "StateServiceDB" -DatabaseServer $databaseServerName | New-SPStateServiceApplication -Name $stateSAName | New-SPStateServiceApplicationProxy -Name "$stateSAName Proxy" -DefaultProxyGroup > $null




Write-Host "Creating Secure Store Service and Proxy..."
New-SPSecureStoreServiceapplication -Name $secureStoreSAName -Sharing:$false -DatabaseServer $databaseServerName -DatabaseName "SecureStoreServiceAppDB" -ApplicationPool $saAppPoolName -auditingEnabled:$true -auditlogmaxsize 30 | New-SPSecureStoreServiceApplicationProxy -name "$secureStoreSAName Proxy" -DefaultProxygroup > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Secure Store Service"} | Start-SPServiceInstance > $null




Write-Host "Creating User Profile Service and Proxy..."
$userProfileService = New-SPProfileServiceApplication -Name $userProfileSAName -ApplicationPool $saAppPoolName -ProfileDBServer $databaseServerName -ProfileDBName "ProfileDB" -SocialDBServer $databaseServerName -SocialDBName "SocialDB" -ProfileSyncDBServer $databaseServerName -ProfileSyncDBName "SyncDB"
New-SPProfileServiceApplicationProxy -Name "$userProfileSAName Proxy" -ServiceApplication $userProfileService -DefaultProxyGroup > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "User Profile Service"} | Start-SPServiceInstance > $null




Write-Host "Creating Visio Graphics Service and Proxy..."
New-SPVisioServiceApplication -Name $visioSAName -ApplicationPool $saAppPoolName > $null
New-SPVisioServiceApplicationProxy -Name "$visioSAName Proxy" -ServiceApplication $visioSAName > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Visio Graphics Service"} | Start-SPServiceInstance > $null




Write-Host "Creating Web Analytics Service and Proxy..."
$stagerSubscription = "<StagingDatabases><StagingDatabase ServerName='$databaseServerName' DatabaseName='StagerDB'/></StagingDatabases>"
$reportingSubscription = "<ReportingDatabases><ReportingDatabase ServerName='$databaseServerName' DatabaseName='WarehouseDB'/></ReportingDatabases>" 
New-SPWebAnalyticsServiceApplication -Name $WebAnalyticsSAName -ApplicationPool $saAppPoolName -ReportingDataRetention 20 -SamplingRate 100 -ListOfReportingDatabases $reportingSubscription -ListOfStagingDatabases $stagerSubscription > $null  
New-SPWebAnalyticsServiceApplicationProxy -Name "$WebAnalyticsSAName Proxy" -ServiceApplication $WebAnalyticsSAName > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Web Analytics Web Service"} | Start-SPServiceInstance > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Web Analytics Data Processing Service"} | Start-SPServiceInstance > $null



Write-Host "Creating Word Conversion Service and Proxy..."
New-SPWordConversionServiceApplication -Name $WordAutomationSAName -ApplicationPool $saAppPoolName -DatabaseServer $databaseServerName -DatabaseName "WordAutomationDB" -Default > $null
Get-SPServiceInstance | where-object {$_.TypeName -eq "Word Automation Services"} | Start-SPServiceInstance > $null

Waiting for My Royalties From Pixar

Wed, 21 Apr 2010 07:21:00 -1000

I spent some time recently in Microsoft Studios recording SharePoint videos which will soon be released to TechNet. During the review of the videos a friend of mine, Vesku, pointed out one video in which my expression was a bit odd. Taking a look at the image I immediately discovered an unfortunate truth – I look like Carl Fredrickson…

An Expected Error Has Occurred

Wed, 07 Apr 2010 16:55:00 -1000

There is not a single sole on this planet which has used SharePoint that has not seen the error message “An Unexpected Error has occurred”. Bold statement, and obviously I don’t have the research to back it up but you must admit its an error allot of folks see from time to time if you spend any time on a SharePoint site – hopefully that will change soon.

With SharePoint 2010 you can now set your own custom error pages. This has been a much asked for feature from allot of my customers and has the potential to offer a much better user experience. In fact, one which you may choose to vary depending on the role of the user. For example, you may want to provide a very generic error for anonymous users and a much more detailed error of the user happens to be the site collection owner or farm admin.

Its not only the error pages that got all the love here, in fact you can customize the Access Denied, Confirmation, Login, Request Access, Sign-out, and Web Deleted pages. The SPWebApplication.SPCustomPage enumerator is used to pass the page which you wish to override into a call to UpdateMappedPage() along with the path to the custom page. So once set, this page will be used anytime a request is made for the corresponding page. Since this method is part of the SPWebApplication you will note a custom pages are scoped to the web application.

What is really happening here is SharePoint’s Virtual Path Provider (VPP) does a check each time it’s asked to resolve a path to a file and if the current Web Application has a custom page configured it will be served rather than the original error page. This means URLs and code references to these pages will not need to be updated to paths to your custom files because the VPP handles the redirection.

If you choose to remove a custom page you make the same call into UpdateMappedpage passing NULL for the page path and the corresponding enum value for the page you wish to disconnect.

I would highly recommend anyone who deploys an Internet facing SharePoint site override at the very least the error page for at least three reasons:

1) Own your messaging to the anonymous user, don’t let SharePoint send messages to your users. What is the point of giving them an error they most likely cannot understand, don’t care too, and most likely could not take any corrective action to fix.

2) Currently there is a bug that will hopefully get fixed before SharePoint 2010 hits RTM which allows anyone to place an error message on the error page, for example on your SharePoint site append this string to your URL: _layouts/error.aspx?ErrorText=ErrorBaby!

3) There is an opportunity to offer a better user experience, the custom error page might collect additional information and send that along to those interested, you may choose to collect additional information from the user by asking for feedback, or you may offer additional details if the user is privileged enough, based on his or her role to understand what may be going on with the server.

Here is some example code to get you started – this is a custom feature receiver which you might use to hook a custom error page.

  [Guid("bf07336a-76d7-4324-8908-74e2771bba49")]
   public class CustomPageMappingsFeatureEventReceiver : SPFeatureReceiver
   {
       const string CustomErrorPage = "/_layouts/CustomPageMappings/Error.aspx";
       
       public override void FeatureActivated(SPFeatureReceiverProperties properties)
       {
           SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;
           if (webApp != null)
           {
               if (!webApp.UpdateMappedPage(SPWebApplication.SPCustomPage.Error, CustomErrorPage))
               {
                   throw new ApplicationException("Cannot create the new error page mapping.");
               }
               webApp.Update(true);
           }
       }
       
       public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
       {
           SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;
           if (webApp != null)
           {
               if (!webApp.UpdateMappedPage(SPWebApplication.SPCustomPage.Error, null))
               {
                   throw new ApplicationException("Cannot reset the default error page mapping.");
               }
               webApp.Update(true);
           }
       }
   }

Expanding the Refiners

Mon, 05 Apr 2010 05:58:00 -1000

SharePoint 2010 Search has many new features but one of which will be most evident to users will be the inclusion of a new WebPart called the Refinement Panel. This WebPart allows the user to refine the search results returned from a query even further by viewing a list of refiners (categories). The user may choose to reissue the query with the refinement to filter the results further. For example, lets say your query returned 100 documents which were authored by 5 different people within your organization. The out of the box (OOTB) the Refinement Panel has an Author refiner and it would show you the 5 Author’s names, which in itself is very interesting, but in addition you could select an author and get a list of just those documents authored by that particular author.

OOTB SharePoint has six refiners or “Categories” including Author, Result Type, Site, Modified Date, Managed Metadata columns, and Tags. Each is used in much the same way, as I explained previously in my Author example, however each differ in what data they look at to determine a grouping to refine upon as well as how the query is reformatted to refine the results.

In this post I would like to show you how to add your own Category without having to write any code (this assumes you don’t consider XML code). In our scenario we want to add a Category which allows us to refine the search results based on the size of the document. That is, we want to filter or refine our search results based on the size of the documents as returned from issuing a query. For this to work I need to create a classification system for my documents based on their size. The table below is what I am using to group my documents; obviously the name and value ranges are of my choosing and you can choose any which make since to your scenario but this is what we will use in this example.

Name	Min value	Max value
Tiny	0	10,000
Small	10,001	100,000
Medium	100,001	1,000,000
Large	1,000,001	5,000,000
Huge	5,000,000	Infinite

The easiest way to get started is by creating a new web based on the Search Center template provided by SharePoint 2010. Once created, navigate to the results page (results.aspx) and put the page into edit mode. On the left side of the page you will find the Refinement Panel Web Part which we need to edit. Once the Editor Panel open navigate to the “Refinement” section and expand. There are 3 items we want to change. The first change we need to make is to the Filter Category Definition; this is a XML document the Refinement Panel Web Part uses to define the categories which represent the various refinements. The second is the Number of Categories to Display. This option allows you to choose the max number of categories which will be displayed when a Refiner is available. Categories are only displayed when there are items found which can be refined further. For example, the Tag refiner will only be displayed if there are documents, within the search results, which have been tagged. Since this example adds a 7th category its important we change this value to 7. The last configuration change we need to make is to uncheck the Use Default Configuration checkbox. With this selected any changes to the Filter Category Definition will not be persisted. Its unfortunate this checkbox did not just disable the Filter Category Definition configuration UI but let me just save you some time and tell you that if you plan on editing the Filter Category Definition you will need this checkbox unchecked.

So lets take a look at the Filter Category Definition XML document. The document has a root element of FilterCategories with Category sub elements which each represent the refiner categories. Reading through the document’s default content offers a great way to learn the syntax and provides examples. The nodes and attributes are actually pretty well named such that you can get an idea pretty quickly what is going on. The Type attribute is very important and points to a class which implements the filter called the filter generator. OOTB there are three including ManagedPropertyFilterGenerator (which we use here), TaxonomyFilterGenerator, and RankingModelFilterGenerator (digging into each of these filter generators is beyond the scope of this blog post). If none of these filters are what you want you can write your own by deriving from RefinementFilterGenerator and adding your own implementation. If I can think of a scenario where this might be needed I may do another blog post on this later.

In this example I am going to add a new category named “File Size” for which I have included the definition below. Since this is based on the managed property filter generator its important that we define the MappedProperty, in this case, “Size”, which represents the file size of the document. I include a CustomFilters element and set the DataType to Numeric (String and Date are acceptable here too) which properly represents the type coming from the managed property stored in Search. Next I implement the size ranges/groups, which I detailed earlier in the table above, using the format “A..B” where A and B are optional but one is always required. The formatting rules are, A must be less than B when B is included, and B must be greater than A when A is included. So for my first filter I called it “Tiny” and gave it no lower bound but an upper bound of 10,000. For “Small” I used similar syntax and defined the range from 10,0001 to 100,000.

<Category Title="File Size"
            Description="The size of each document"
            Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"
            MetadataThreshold="2"
            NumberOfFiltersToDisplay="6"
            MaxNumberOfFilters="0"
            SortBy="Custom"
            ShowMoreLink="True"
            MappedProperty="Size"
            MoreLinkText="show more"
            LessLinkText="show fewer">
    <CustomFilters MappingType="RangeMapping" DataType="Numeric" ValueReference="Absolute" ShowAllInMore="False">
      <CustomFilter CustomValue="Tiny">
        <OriginalValue>..10000</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Small">
        <OriginalValue>10001..100000</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Medium">
        <OriginalValue>100001..500000</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Large">
        <OriginalValue>500001..2000000</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Larger">
        <OriginalValue>2000001..5000000</OriginalValue>
      </CustomFilter>
      <CustomFilter CustomValue="Huge">
        <OriginalValue>5000001..</OriginalValue>
      </CustomFilter>
    </CustomFilters>
  </Category>

With the new XML created we can edit the Filter Category Definition and append our XML fragment into the document ensuring the XML is still well-formed (if we fail to format the XML properly the Web Part will tell us before we exit edit mode, so be on the lookout for that message).

After leaving edit mode submit a query to search and see the results. Below is a screenshot of what we just created with the new category highlighted in yellow. Note that if I click on any of the filters such as Tiny, Small, or Medium I will see search results based on those filters. In addition you will note we do not have filters for “Large” and “Huge” as we defined earlier, this is because our results set does not include any documents which have sizes which fall into these ranges.

Search refiners are a powerful way to allow uses to dig deeper into the query results and find what they are looking for. Couple that with the ability to customize the refiners on your site and you have a powerful feature which I am sure your users will find very useful.

In Pays To Keep an Eye on Your (Object) Cache?

Mon, 15 Mar 2010 17:14:00 -1000

Since MOSS 2007 publishing sites have had a special type of cache known as the object cache. This cache is used to speed up navigation, cross-list queries, and other publishing operations by not constantly having to hit the DB for each request. The cache is scoped at the site collection and Site Collection Administrators have the ability to change several settings related the cache’s operation. One in particular is the max amount of memory which SharePoint will allow the cache to grow. Playing with this setting you may be surprised to learn SharePoint allows this value to range from 1 MB up to 10 TB! Yep, SharePoint allows Site Collection Administrators with a publishing site to set their object cache up to 10 TB for each Site Collection for which they are an administrator.

Here is a setting on my server – not seeing any performance gains yet but still testing. :-)

Its not all bad news here; in SharePoint 2010 we do have a web.config setting which helps control the total amount of memory dedicated to object cache per Web Application however I still question why Site Collection administrators are allowed to set this value. In almost all cases, a Site Collection administrator does not have the means to determine the proper setting. They don’t have performance monitor, access to the IIS Logs, knowledge of what other Site Collections are running on the machine, performance characteristics of the web application, or insight into what hardware, specifically the amount of memory is available on the WFE servers.

So the moral of the story? Keep your friends close, but your Site Collection Administrators closer.

SharePoint’s Sasquatch Memory Leak

Mon, 08 Feb 2010 10:26:00 -1000

Background

I have been working with SharePoint for a few years now and have run into many nasty high memory or Out of Memory (OOM) issues. To date many of SharePoint’s memory problems have been discussed as problems with developers not using the Dispose pattern properly when using the SharePoint OM. And while I have found this to be the case I always had a feeling that there was something more, another larger leak which could not be explained simply by not calling Dispose. Sure not calling dispose will keep the SPRequest COM object (which is fairly large) around longer than necessary however ref-counting will clean these up eventually but even with using the correct coding patterns and practices the w3wp processes still seem larger than necessary; even on my development machine when I run stress against an out of the box installation. So there must be something else going on here; and so started my search for Big Foot.

Big Foot Makes an Appearance

Recently I was contacted by a friend who works on the IIS team to help troubleshoot a memory issue with SharePoint. This is a pretty common request but what really piqued my interest was when he told me he had a 9 GB memory dump. For those not familiar with memory dumps the size represents the size of the memory being used by the process so for this w3wp process it was using 9 GB of memory and to make matters worse the process had only been up about 56 min. This seemed like the best opportunity to find a nasty and elusive memory leak!

Cracking open the memory dump I found something very interesting, the process had upward of 24,600+ ASP.global_asax objects. So why is this important? The global_asax represents the file based HttpApplication which is used during request processing. You can think of these guys as the tour guide to the user’s request which guides the request through ASP.net’s request processing pipeline. The HttpApplicationFactory keeps a free list of HttpApplication instances and as new requests reach the system one is pulledf from the free list and assigned to the request to guide the request through the pipeline or in the case when none are available will instantiate a new HttpApplication. Once complete the HttpApplication is either returned to the free list or is discarded depending on the number of HttpApplication(s) already in the free list. In addition the HttpApplicationFactory will also attempt to trim unused or unneeded HttpApplications from the free list every 30 seconds. For each pass which a free HttpApplication exists within the free list a single HttpApplication will be disposed.

So now that we understand HttpApplication lifetime why is it that this process memory dump have so many? Looking at the HttpApplicationFactory I found that only about 82 HttpApplications were in the free list and combined with the number which were processing requests at the time of the memory dump we can only account for less than 90. This means the other 24,000+ were either being leaked or have yet to be collected by the GC. Since I did not believe that the lack of GC to be the problem I started to look for what was keeping these objects rooted such that the GC could not collect them. I soon came across this object:

0:000> !do 00000002435a8290

Name: System.UnhandledExceptionEventHandler

MethodTable: 0000064278467860

EEClass: 0000064278005670

Size: 64(0x40) bytes

GC Generation: 2

(C:\WINNT\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:

              MT            Field           Offset                 Type VT             Attr            Value Name

0000064278435ed8  40000ff        8        System.Object  0 instance 00000002435a8290 _target

0000064278434990  4000100       10 ...ection.MethodBase  0 instance 0000000000000000 _methodBase

0000064278438ff0  4000101       18        System.IntPtr  1 instance 6882685329544 _methodPtr

0000064278438ff0  4000102       20        System.IntPtr  1 instance 6882550903584 _methodPtrAux

0000064278435ed8  400010c       28        System.Object  0 instance 0000000193373b88 _invocationList

0000064278438ff0  400010d       30        System.IntPtr  1 instance 24686 _invocationCount

0:000> !do 00000002435a8290
Name: System.UnhandledExceptionEventHandler
MethodTable: 0000064278467860
EEClass: 0000064278005670
Size: 64(0x40) bytes
GC Generation: 2
(C:\WINNT\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
              MT            Field           Offset                 Type VT             Attr            Value Name
0000064278435ed8 40000ff        8        System.Object 0 instance 00000002435a8290 _target
0000064278434990 4000100       10 ...ection.MethodBase 0 instance 0000000000000000 _methodBase
0000064278438ff0 4000101       18        System.IntPtr 1 instance 6882685329544 _methodPtr
0000064278438ff0 4000102       20        System.IntPtr 1 instance 6882550903584 _methodPtrAux
0000064278435ed8 400010c       28        System.Object 0 instance 0000000193373b88 _invocationList
0000064278438ff0 400010d       30        System.IntPtr 1 instance 24686 _invocationCount

This object is the event handler for an AppDomain’s UnhandledExceptionEventHandler. Note the _invocationCount member, its value is 24,686. What this means is that over 24,000 objects have registered for notification when the AppDomain for which this object exists exits abnormally. Digging further I found the SPHttpApplication as the type which the _target was referencing. This object derives from HttpApplication and is itself derived from ASP.Net’s generated global_asax class. In SPHttpApplication’s Init() method it hooks the AppDomain’s UnhandledExceptionEventHandler. While the callback for this event logs what may be important information to the ULS log when an unexpected process exit it does not need to do this 24,000+ times.

As most readers will already know static objects live for the life of the AppDomain so an AppDomain’s lifetime is equal to that of static objects within the .Net Framework. As such when an object registers for event notification from an object which is either static (or in this case the AppDomain) the registering object roots itself for the lifetime of the AppDomain. The GC cannot collect this object until either the AppDomain restarts or the object unregisters itself from the event handler. Unfortunately SPHttpApplication does not unregister itself from the AppDomain’s UnhandledExceptionEventHandler handler so each instance will live as long as the AppDomain. So even after the HttpApplicationFactory releases the SPHttpApplication the object will continue to remain in memory until the AppDomain restarts.

So we have a leak, however it gets worse..the ASP.Net runtime creates a new HttpContext for each request which is processed. Its this object which holds references to the HttpRequest, HttpResponse, and most importantly the handler which will service the request. For the cases where this handler is a page the entire control hierarchy, member variables, etc is referenced. Since the HttpApplication indirectly has a reference to the HttpContext the amount of memory being referenced can be huge. Take a look at these objects which I found in the memory dump, note that for each HttpApplication which is created an HttpContext, HttpResponse, etc are too created.

Number of instances	Total Size	Type
35,449	1,227,120	System.String[][]
35,449	1,417,960	System.Web.Security.CookielessHelperClass
35,449	2,268,736	System.Web.HttpResponseStream
35,449	5,104,656	System.Web.HttpWriter
35,449	11,343,680	System.Web.HttpResponse
35,449	11,910,864	System.Web.HttpRequest
35,449	11,910,864	System.Web.HttpContext

So what’s the bottom line? – For each HttpApplication which is leaked a potential huge number of child references are too rooted and therefore leaked. While the table above would seem to indicate this is a small leak the numbers above do not reflect the size of all child objects which have been rooted.

So What is The Fix?

As of today we have just begun the process of having this issue addressed by the SharePoint 2007 servicing team so we do not have a fix at the time this blog was published. Once the fix is available I strongly advise everyone to install it and if implemented remove the workaround I provide below.

Workaround Please

For those that do not want to wait for a fix I have a workaround however it will require a bit of coding on your part but its really not that difficult. What we want to do here is ensure we only let the SPHttpApplication register only once for the AppDomain’s UnhandledExceptionEventHandler. Since the SPHttpApplication does this from within its Init() virtual method what we can do is ensure this gets called only once so as to only root one HttpApplication instance.

Within VS2008 create a new class library and add a reference to the Microsoft.SharePoint library. Ensure you also strong name the assembly because we want to place this assembly in the GAC.

Add a new class and paste the code below into this file.

using System;
using Microsoft.SharePoint.ApplicationRuntime;

namespace ToddBlog.Sample
{
    public class CustomHttpApplication : SPHttpApplication
    {
        private static bool s_initialized = false;

        public CustomHttpApplication()
        {
        }

        public override void Init()
        {
            if (!s_initialized)
            {
                lock (typeof(CustomHttpApplication))
                {
                    if (!s_initialized)
                    {
                        base.Init();
                        s_initialized = true;
                    }
                }
            }
        }
    }
}

Compile the project and deploy the resulting assembly into the GAC on every SharePoint WFE server within your farm.
For each web application edit the global.asax file and change the assembly reference to point to your custom assembly created in step #1 and change the Inherits attribute to reference the class you created in Step #2.

<%@ Assembly Name="[name of your assembly"]%>
<% Application Language="C#" Inherits="ToddBlog.Sample.CustomHttpApplication" %>

Results

After having the customer install the fix I have described here the process which would normally grow to 10 GB an hour and recycle never went above 2.5 GB! And even after taking all servers out of the load balancer and running the entire farm off one WFE the memory never went above 3.3 GB!!

Who’s Impacted

This issue is applicable to MOSS 2007 and WSS 3.0. SharePoint 2010 is not effected by this issue. The impact to your application depends on many factors such as concurrency, traffic and the amount of uptime. For most decent size installations however this could greatly reduce the amount of memory needed by the IIS Application pool process.

Feedback

At the time of this post I have only worked with one customer for which this fix was huge. I would love to hear from others which implement this fix to hear how much impact it had in other environments, so please leave a comment with your results.

New Home

Mon, 08 Feb 2010 06:34:00 -1000

Been meaing to move my blog onto its own site. Yea I don't do much blogging but I did want to move off MSDN just in case I start up again.