Adventures In SharePoint Land

SharePoint Saturday Charlotte

jefferydalton@hotmail.com (Jeff Dalton) — Mon, 22 Jun 2009 00:54:00 +0000

This past Saturday I had the privilege of presenting at SharePoint Saturday Charlotte Event along side some top talent in the SharePoint community. It was great to finally meet some of the people I follow on Twitter (too many to name). Also, big kudos to Dan Lewis @danlewisnet, Brian Gough and all of the #SPSCLT Volunteers, you guys/girls rock!!!

As promised here the slide deck from my talk about Performance Testing with SharePoint. I was hoping for a little bigger turnout, but when I saw I was in the same time slot as Becky Isserman (@MossLover) and Laura Rogers (@WonderLaura) I knew I would be lucky to get 7 people. :)

I really enjoyed the sessions I got to attend.

The day started off with a GREAT presentation from Phil Wicklund. Phil had lots of pragmatic advice for managing your SharePoint investment.

Next, I learned a LOT about how MS has implemented Windows Azure from Rick Taylor (@slkrck). Rick is a GREAT speaker and had lots of war stories (which I LOVE hearing).

Then, I got to hear Mike Watson (@mikewat) talk about SharePoint hosting architectures. He has some really good insight into what it takes to make SharePoint purr. Also, I got to hear that our hosting architecture for SharePoint is in line with what he thinks is the RIGHT way to do it.

Next, I listened into Dan User (@usher) talk about Taxonomies. I felt much better about my Internet facing solution that will have about 30 Web Applications in one Farm after hearing what he is doing.

Finally, I listened to Dan Attis (@jdattis) talk about a solution he recently wrapped up that used SharePoint lists to store data for a Web Interface that was not SharePoint. It sounds like a really cool solution the folks at B&R built. Also, I learned about Object Initializers in .Net 3.5, really cool stuff.

Definitely will not be my last SharePoint Saturday event.

Technorati Tags: SharePoint,SharePoint Saturday

Big Thanks to Office / SharePoint Teams for TAP Airlift

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 05 Jun 2009 01:42:00 +0000

I had the privilege of attending the Office 14 TAP Air Lift this week in Seattle. This is my second time coming out to to a SharePoint Air Lift and I must say they never disappoint. While I cannot share any information from the Air Lift I can say it is exciting times to be working with SharePoint.

I really want to extend a big thank you to Microsoft for hosting a great event. The folks in Office / SharePoint development teams have some big deadlines in front of them. For them to take time out of their busy schedules to spend some one on one time with customers says a LOT.

More Lessons learned from Performance Testing SharePoint

jefferydalton@hotmail.com (Jeff Dalton) — Sat, 16 May 2009 15:21:00 +0000

Abstract

Performance testing with SharePoint, or any web based application, can be quite tricky. Recently my team launched an upgraded Corporate Web Site based on SharePoint 2007. The launch was quite challenging mainly due to mistakes made during performance testing Lessons Learned from Intranet Launch.

This post is dedicated to the lessons learned from the performance testing of Corporate Web Site.

Background

Prior to launch we ran through our performance test scenarios 3 times. Each time the output showed that we could scale way beyond the existing implantation of our Corporate Web Site (Referred to as Violin from here on).

The performance test scenarios had been chosen based on traffic patterns and pages determined to be high risk for performance (This was good).

Our key performance requirements stated that the web servers must support 38 page views / sec with response time < 5 sec (This was good). This is a nice well defined requirement, although some could argue that 38 page views needs to be broken down into specific types of pages (ex. 10 home page views, 7 chapter page views, …).

We also had a performance goal stating that processor utilization should not go above 80% on web servers for more than 5 seconds (This was good).

For the final test we replayed traffic from IIS logs that were taken during peak traffic window (when we received the most requests / sec). This was a bit tricky because my Load Runner resource told me that this was not supported by Load Runner. So he and I had to message the data inside the IIS logs to get it so Load Runner would support running the tests (this felt wrong at the time, but I cannot say if it is a mistake).

We used Load Runner (sorry I do not know version) for all of the performance tests. The Load Runner clients were located within the same data center as our web servers, but they were on different network segments.

When we ran the tests we engaged several people from operations team (Network, Windows Server, SQL Server DBA and SharePoint Admin). These people were tasked with monitoring components related to their area of expertise. They were also required to collect performance statistics and report those back so they could be included in overall performance test report (This was good).

“Performance is exceptional” or The False Sense of Security

So each time we ran the tests we were able to reach levels of about 90 page views / sec on one server with avg. response time < 5 seconds (we have 4 load balanced WFE in our farm). So we were hi-fiving and slapping each other on the back. As far as we were concerned performance requirements were met, check them off we are done.

We did notice an occasional spike w/ CPU, but we were able to correlate this back to pages expiring in Output Cache. So this was not a concern.

Well once we went live we discovered that something was gravely wrong.

So What Went Wrong

After going live we discovered that the output cache hit ratio was not aligned with the numbers we were seeing during performance testing. So were were having a LOT less output cache hits. This resulted in the servers having to do a lot more work than originally anticipated.

What could have happened? We thought we did everything right with the performance tests. What went wrong?

Well after much soul searching (and re-reading basics of performance testing) it hit me. "

Oh $hit we didn’t model user variations and think times.

Does that really matter?

Yeah it does, the reason is because we ran a high number of requests but the proportion of cached requests vs. un-cached requests was out of balance. Had we have taken into consideration user think times and other variations(browser type, user location) we would have less hits against output cache.

Classic 101 Performance Testing Mistake. Oh well, you pick yourself up, dust yourself off and vow not to make the same mistake again.

Lessons Learned Summary

1. Think times matter

User think times are critical when doing performance testing (especially for web applications that rely on ASP.Net Output Caching to meet performance goals).

2. End user variations matter

Just as important as think times you need to look at the IIS Logs (or your web analytics reports) to understand browser differences and local differences. This is extremely critical if you have Output Cache configured so it treats these differences as non cached page requests.

3. Mix up the IP addresses to fool user affinity

While this is not as important as Think Times and End User variations it is important if you are doing performance testing through a load balancer configured with session affinity.

All of the tests we ran looked like they were coming from 2 IPs. While I cannot prove this invalidated the test results it looks like there was some sort of caching efficiencies realized somewhere in the stack (Switch, NIC, IIS, …).

References

Microsoft Patterns and Practices: Performance Testing Guidance for Web Applications

Microsoft Office Server Online: Configure page output cache settings

MSDN: Output Caching and Cache Profiles

Technorati Tags: SharePoint,Performance Testing,Load Testing,Lessons Learned

Lessons Learned from Launch of Intranet on SharePoint 2007

jefferydalton@hotmail.com (Jeff Dalton) — Mon, 04 May 2009 00:55:00 +0000

Abstract

This post provides some lessons learned from the launch of our Corporate Intranet. After about two weeks of poor performance and stability issues we stabilized the site and resolved most of the issues. The lessons learned here are common and I'm sure our team was not the first (nor the last) to make these mistakes.

Background

Our Corporate Intranet supports about 21 Business Area / Business Units (BA/BUs). When I say Intranet I am referring to a content publishing web site that provides announcements, latest news, corporate policies and other information that is important for employees to consider. It is not a place for employees to collaborate as teams, this is done by another set of SharePoint 2007 sites.

The Intranet is hosted on something we call the Common Web Platform (CWP). What makes it common is it is one set of features / functionality that powers Intranet, Extranet and Internet publishing sites.

In 2007 I started working on a project to upgrade CWP from its current infrastructure (SharePoint 2003 / MS Content Management Server 2002) to SharePoint 2007. The first major component to rollout under the new SharePoint 2007 version of CWP is the Intranet site.

Our intranet is not small. It contains approximately 67,000 webs, 65,0000 documents and 70,000 web pages. The business requirements for sharing content between BA/BUs led us to determine that putting all this content in one site collection was the best choice. I still believe this was the right decision, but it did cause us to create a SharePoint Content DB that is around 330 GB.

Launch Day

Launch day was actually quite calm from my perspective. Yes we had a large site, but I felt we had done an excellent job with performance testing so launch would actually go quite smooth I do not want to go into specifics but the performance testing done had shown that the new SharePoint 2007 site would be able to scale about 3 X higher in number of page views and users than the existing platform.

Everything wasn't perfect, in fact far from it. We had quite a number of lingering issues from content migration. We also had some application bugs that just would not go away. But everyone agreed that these could be solved so we decided to go forward with the launch.

So at approximately noon Eastern US time on March 25th, 2009 we had the DNS team flip the switch and all traffic rolled off the old environment and to the new. It was one of the smoothest cutover’s I have ever been associated with, I even heard some people saying that the did not know we had flipped the switch.

The next morning as Europe came online the proverbial $hit hit the fan. I'm not going to go into the blow by blow details, but I will say a dedicated team of engineers that wanted nothing more than to see this new platform succeed went to work along with MS Premier Support. On Tuesday April 7th the task force was closed as everyone agreed that while the new Intranet had some problems it was stable and performance was acceptable to end users.

This was a really tough one to troubleshoot. The thing that made it tough was just inconsistency with the crashes. We could never tie it back to one specific event or one set of clear patterns. The only consistency was the fact that it crashed during peak traffic loads (from 2 AM - 9 AM Eastern US time). The Intranet availability dipped to about 60% during these two weeks.

Lessons Learned

As I stated after about two weeks of pure hell we got things stable. During that time we did a lot of analysis and a few changes. So in no particular order here is the things we changed and why and what I personally learned.

1. Hosting web services that do not use SharePoint inside your SharePoint Application Pool is bad (umm kay… South Park ref).

One of our field controls makes calls to a web service that in turn makes calls to a database to retrieve some data. It is pretty basic stuff. Well, to make a long story short the web service ended up in our SharePoint solution package and our field control ended up a call back into the same Application Domain to call some data from a database. Yes, I know not a very smart thing to do.

Anyway, during the performance we put together specific KPI’s to watch for this web service. We saw no major problems with it, but put it on a list of things to change once the application went into maintenance mode.

While we never linked any outages specifically to calls to this web service, however we did see a major improvement in overall stability when the web service was moved to a separate application pool.

So the lesson learned is to keep the Application Pools that host SharePoint sites dedicated to SharePoint sites (do not have those Application Pools host non SharePoint IIS Sites).

2. Be sure to set RowLimit on query at less than 5000 items to avoid table locks.

One of the problems that definitely caused outages was table locks at the SQL Server level. We traced the table locks back to SQL that was being generated by a CAML query we used to show documents associated with a given web page.

SQL will lock a table if it thinks a query will return more than 5000 rows. So it is very important that you set a row limit when using SPQuery and CrossListQueryCache objects. When SharePoint generates the SQL for CrossListQuery if will set a default row limit of 2 million items. I’m not sure if it does the same thing for SPQuery, but better safe than sorry.

So the lesson learned here is always set a row limit that is less than 5000 when using SPQuery and CrossListQueryCache.

3. If querying by FileRef use SPWeb.GetListItem instead

The CAML Query referenced in Item 2 above was using FileRef field to filter the result list. Unfortunately FileRef is a special field inside of SharePoint, meaning it doesn’t lend itself to be indexed (See Index List Field). So the SQL query’s that were generated from the CAML were doing full table scans which is another big performance hit and can cause unwanted database locks.

So in the end we abandoned using CAML query to get the documents and instead pulled them the SPWeb.GetListItem method. At first there was a hugh debate on our team, because fundamentally it is better to reduce communication with DB. So we were going from essentially one call to the DB to two calls per file in our document list field control (note: SPWeb.GetListItem results in at least 2 calls to the DB, one to get the List field info and one to pull the ListItem data).

Our control has a limit of 200 documents that can be displayed. So we knew the maximum number of times we would call GetListItem per page would be 200. We also knew that the average number of documents per page was 3. So most pages had very few documents to display.

Our team is looking at alternate approaches. One idea is to add a field to each document that has a GUID. Then index that field and go back to doing queries using that new field. We have a lot of testing to do before we make a decision to go in that direction.

So the lesson learned was do not write CAML query's that use FileRef as the primary field to filter the results.

4. Don't make these mistakes with performance testing.

Okay this one requires a separate blog post. I promise to post a blog entry with this information very soon. In the mean time I can say that the key mistake made with performance testing was not taking into consideration user sessions and think times. We had the right URLs (we took these straight from logs of production machine), but we ran them through too fast which created a situation where URLs uses output cached versions of the pages when under normal load they would not have used the cached versions.

Wrapping Up

Granted we had a rough launch because the performance testing did not catch the critical application issues. I do not want to leave people with the impression that everything we did was wrong. Our team did a lot of stuff write and often these things get forgotten when things go wrong. So here is a short list of the things we did right:

We used 64 bit hardware for all our servers (SQL and Web Front Ends).
We used the caching options with Publishing sites effectively (Output Cache, Object Cache and BLOB Cache).
We discovered a major memory leak in our code with performance testing and fixed it before going live.
We put together a well defined set of Solutions and Features for our application (so we can deploy easily).
We created a team of people that have some really deep knowledge on building SharePoint Publishing sites.

References

MSDN: Best Practices: Common Coding Issues When Using the SharePoint Object Model

Microsoft TechNet: Tune Web server performance (Office SharePoint Server)

SharePoint for End Users: Manage large SharePoint lists for better performance

Reza Alirezaei’s Blog: 20 key Points Arising, or Inferred, From “Working with large lists in MOSS 2007” Paper

Technorati Tags: SharePoint,Development

I'm Back...

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 01 May 2009 00:01:00 +0000

It's been a while since my last post.

I've been heads down in the middle of our corporate rollout of SharePoint 2007. Specifically I have been working on project to upgrade our content management system from MCMS 2002 / SharePoint 2003 to SharePoint 2007. Exciting stuff and every day presents new challenges.

We just recently went live with our Intranet and now are starting work on Extranet and Internet sites.

Managing Key-Value Pair Settings in SharePoint

jefferydalton@hotmail.com (Jeff Dalton) — Mon, 02 Jun 2008 02:29:00 +0000

Recently my team discussed various ways to manage application configuration settings in SharePoint. We wanted to avoid using Web.Config files for Key-Value Pair data because of the complexity of managing this data in SharePoint.

SharePoint contains a decent API (SPWebConfigModification) for managing web.config settings across the farm. It works pretty good, but we have had some trouble with it in certain situations. If you are interested in a good article on SPWebConfigModification check out this one by Mark Wagner

Thankfully the folks on the SharePoint development team delivered just what we needed to have a simple solution for managing key-value pair settings. It is the SPPropertyBag class.

SharePoint provides a properties collection for SPFarm, SPWebApplication, SPWebService and SPWeb objects (none for SPSite, but site settings can be stored @ SPSite.RootWeb).

The properties collection for SPFarm, SPWebApplication and SPWebService are not really based on the SPPropertyBag class. The properties collection comes from the derived class SPPersistedObject and is actually a HashTable. But it works exactly the same as SPPropertyBag.

It is really easy to manage the properties settings. Here is some sample code to update settings for SPFarm, SPWebApplication or SPWebService.

public static void SetPropertyValue(SPPersistedObject spObject, string name, string value)
{                   
    if (spObject.Properties.ContainsKey(name))
    {
        spObject.Properties[name] = value;
    }
    else
    {
        spObject.Properties.Add(name, value);
    }
    spObject.Update();
}

The SPWeb class actually uses SPPropertyBag for its properties. The code to manage SPWeb.Properties is similar.

public static void SetPropertyValue(SPWeb webSite, string name, string value)
{
    if (webSite.Properties.ContainsKey(name))
    {
        webSite.Properties[name] = value;
    }
    else
    {
        webSite.Properties.Add(name, value);
    }
    webSite.Properties.Update();
}

Technorati Tags: SharePoint,Key-Value Pair Settings,Application Configuration Settings

Creating a SharePoint Web with the API

jefferydalton@hotmail.com (Jeff Dalton) — Wed, 21 May 2008 05:28:00 +0000

I have been working on a application that provides a repeatable way to create sites. This application is very similar to the SharePoint Test Data Population Tool.

One of the first tasks was to create new web applications. A quick look at the API and I thought well this is simple enough.

First instantiate a new instance of SPWebApplicationBuilder. Set some properties on my object and then call create. Use the SPWebApplication.Provision() method and I am done.

Well that is not exactly true. What I discovered was that the Provision method only creates the web application on the local server. This is great for a single web front end, but for a more typical setup more work is required.

So I dug into the Administrative screens using Reflector. I discovered that the SharePoint team is instantiating a timer job that will provision the newly created web on the other web front ends.

if (SPFarm.Local.TimerService.Instances.Count > 1)
{
       SPWebApplicationProvisioningJobDefinition definition = new SPWebApplicationProvisioningJobDefinition(application, this.ApplicationPoolSection.ResetIis);
       definition.Schedule = new SPOneTimeSchedule(DateTime.Now);
       definition.Update();
}

So I thought okay this is simple enough. Then I discovered that SPWebApplicationProvisioningJobDefinition as marked as internal. Agghhhh!!!!

This pattern of finding a useful SharePoint routine only to have it hidden with the Internal keyword happens way too often. I am sure that usually there is a good reason to hide away the functionality, but I could see no logical reason to hide SPWebApplicationProvisioningJobDefinition.

So after throwing a few darts at the SharePoint Development Team dartboard (No I do not actually have one, but it could be a great gift SharePoint developers) I sat down and created a timer job definition that does basically the same thing as SPWebApplicationProvisioningJobDefinition.

This was the first timer job I had ever written so I made a *few* mistakes.

First I failed to understand a basic concept of timer jobs. That is any classes inheriting from SPJobDefinition need to be deployed to the GAC on each SharePoint server. If they are not then the job will not be created correctly. The frustrating thing I encountered was the fact that no runtime error was raised. So it looked like the job definition was created correctly, but it was not.

Second I failed to put a default constructor on my job definition class. This resulted in a runtime error with the job definition. The runtime error indicated that the class failed to serialize properly.

Third I failed to decorate my member variables with the [Persisted] attribute. This resulted in me losing my variable values.

Fourth I did not realize that I have to manually delete the job definitions when they fail. To delete the timer job definition bring up the SharePoint Central Administration web site. On the Operations tab choose Timer job definitions (under the Global Configuration header). Next click the timer job definition in the list and then click the Delete button.

Fifth I failed to understand that job definitions must be uniquely named. This was a problem for me because my application would create one timer job per Web Application. Sometimes a new web application job would be created before the other job was finished. To work around the issue I included a timestamp in the name. I am not sure if this was smart, but it works.

Sixth I failed to understand that the OWSTimer.exe service must be restarted when deploying a new version of my timer job class library. The OWSTimer.exe service is what manages the timer jobs. It actually instantiates and executes the timer job classes. Since my timer job class was deployed to the GAC it meant I had to restart the OWSTimer.exe (on each SharePoint server ughhh!!!).

In the end I accomplished my goal, but it would have been a lot easier if the SharePoint development team had not marked SPWebApplicationProvisioningJobDefinition as internal.

Technorati Tags: SharePoint,Development,Timer Job

What is going on in MOSS Land

jefferydalton@hotmail.com (Jeff Dalton) — Sun, 18 May 2008 17:27:00 +0000

It has been a while since my last post. I have been heads down working on a large project for my employer. The project is to convert my employers existing MCMS/SharePoint 2003 solution to SharePoint 2007. When this project is over my employer will have one of the largest SharePoint sites in the world.

Stay tuned, I am sure there will be many existing adventures to post about.

Adventures in extending the Publishing.Fields.LinkValue class

jefferydalton@hotmail.com (Jeff Dalton) — Sun, 18 May 2008 17:19:00 +0000

Recently I was working on a SharePoint application that required me to extend the out-of-box Publishing Link field. In the end I found an acceptable solution, but it took a little while to get there.

Inheriting from LinkFieldValue

The first thing I tried was creating a new link field value class that extended the LinkFieldValue class. The LinkFieldValue class is not sealed so I thought I would be able to work with it.

LinkFieldValue inherits from the HtmlTagValue class. HtmlTagValue is more or less a dictionary that provides a structured way to create an HTML tag. Unfortunately HtmlTagValue has all of the routines I needed marked as Internal. So this meant that my idea of inheriting from LinkFieldValue was dead.

The Publishing LinkFieldValue class inherits from the Microsoft.SharePoint.Publishing.Fields.HtmlTagValue class. If you look at the guts of LinkFieldValue you will see that HtmlTagValue does all the heavy lifting. The LinkFieldValue properties call back into the HtmlTagValue.Item property.

HtmlTagValue class has two public routines (everything else is Internal or private). So that pretty much ended my idea of extending the LinkFieldValue class.

Storing link data in SPFieldMultiColumnValue field

Next I decided to store the link data in a SPFieldMultiColumnValue field. I have created plenty of these in the past so I knew exactly what to do.

Everything was great until I ran a test for Link fix-up.

In case you did not know WSS has built-in functionality that supports fixing site collection links that get broken because the WSS object (image, document, publishing page) is moved.

Turns out WSS link fix-up only works for certain field types (and Text is not one of them). It also requires that the link be stored as an <A> anchor tag. So that pretty much ended my idea of using the SPFieldMultiColumnValue field.

Using XML to store the Link data

Finally I decided to store the link using the same technique as the Summary Link control. The Summary Link control provides a way of storing a variable number of links in a publishing field. It does this by serializing the data into XHTML and storing it in a rich text field.

First I created a base field class that would act as a parent for any field that I wanted to store as XHTML.

public class XmlAsHtmlField : SPFieldMultiLineText
{
  public XmlAsHtmlField(SPFieldCollection fields, string fieldName) : base(fields, fieldName) { }
  public XmlAsHtmlField(SPFieldCollection fields, string typeName, string displayName) : base(fields, typeName, displayName) 
  {
     base.RichText = true;
     base.SetRichTextMode(SPRichTextMode.FullHtml);
  }
}

Next I created a XmlAsHtmlFieldValue class that would act as a base class for all field values that use the XmlAsHtmlField class. The base field value class implements a dictionary that provides a mechanism for all child classes to store their property values. The class assumes that any properties stored in the dictionary translate their value into XHTML.

public class XmlAsHtmlValue
{
   private Dictionary<string, XmlStorageItem> storageItems = new Dictionary<string, XmlStorageItem>();
       
   /// <summary>
   /// Initializes the XmlAsHtmlValue class
   /// </summary>
   public XmlAsHtmlValue()
   {
   }

   /// <summary>
   /// Initializes the XmlAsHtmlValue class and populates the Dictionary based on the provided XML
   /// </summary>
   /// <param name="HtmlValue"></param>
   public XmlAsHtmlValue(string HtmlValue)
   {
      HtmlValue = Utility.FixQuotesInXML(HtmlValue);

      System.Xml.XmlDocument xmlDocument = new XmlDocument();
      xmlDocument.LoadXml(HtmlValue);
      XmlNodeList storageItemNodes = xmlDocument.SelectNodes("/div/div");
      foreach (XmlNode storageItemNode in storageItemNodes)
      {
         XmlAttribute classValue = storageItemNode.Attributes["class"];
         string id = storageItemNode.Attributes["id"].Value;
         XmlStorageItemType itemType = (XmlStorageItemType)System.Enum.Parse(typeof(XmlStorageItemType), classValue.Value);
         storageItems.Add(id, new XmlStorageItem(itemType, storageItemNode.InnerXml, id));
      }
   }

   /// <summary>
   /// Returns the dictionary that is used to store properties
   /// </summary>
   public Dictionary<string, XmlStorageItem> StorageItems
   {
      get
      {
         return storageItems;
      }
   }

   /// <summary>
   /// Creates an XML representation of the Dictionary
   /// </summary>
   /// <returns></returns>
   public virtual string ToXml()
   {
      StringBuilder returnValue = new StringBuilder();
      returnValue.Append("<div id=\"root\">");

      foreach (KeyValuePair<string, XmlStorageItem> item in storageItems)
      {
         XmlStorageItem storageItem = item.Value;
         returnValue.AppendFormat("<div id=\"{0}\" class=\"{1}\">{2}</div>", storageItem.ID, storageItem.StorageItemType, storageItem.StorageItemValue);
      }
      returnValue.Append("</div>");

      return returnValue.ToString();
   }

   public override string ToString()
   {
      return this.ToXml();
   }

}

One really ugly problem I ran into was the fact that Microsoft will strip out double and single quotes from the HTML. I am not sure why this is done, but I had to create a routine to put the quotes back in.

Next I created an extended link value class that used the out-of-box link field value plus another class I created with the new link properties. I stored both of the classes in the XMLAsHtmlValue class.

Because I used the out-of-box Link Value class I ran into some problems when I created unit tests. When I tried to generate the tests in Visual Studio.Net 2008 I would receive the following error.

Could not resolve member reference: Microsoft.SharePoint.ISPConversionProcessor::PostProcess

To fix the problem I added the following DLL as a reference in my project (Microsoft.HtmlTrans.Interface, it is located in the GAC)

Parting Thoughts

In the end I had something that would allow me to create fairly extensible field value classes. I am not sure how well this solution scales (a topic for a future posting) and I am not how easy this solution will be to upgrade with future versions of SharePoint.

Problems with XMLNS and SPWebConfigModification

jefferydalton@hotmail.com (Jeff Dalton) — Sat, 03 May 2008 01:09:00 +0000

Recently I posted an article that discussed the SPWebConfigModification. I meant to add some information about problems with XML Namespaces and SPWebConfigModification.

One of the first problems my team encountered with SPWebConfigModification was failures when working with configuration settings that overrode the default xml namespace (xmlns).

The problem arose when we tried to apply the webconfig modifications needed to for the Enterprise Library. Below is a sample entry from the configuration settings needed for Enterprise Library. Notice how the default xmlns is being overridden.

<enterpriselibrary.configurationSettings applicationName="NewsApplication" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.microsoft.com/practices/enterpriselibrary/08-31-2004/configuration" />

That one statement invalidates the XPath expressions in the SPWebConfigModification Name and Path properties. This means that SPWebConfigModification cannot successfully find the entry when it scans the web.config file. So every time the web.config file is modified by SPWebConfigModification the entry gets duplicated (which causes the web.config file to be invalid).

A little research into the way .Net Framework handles XML namespaces turned up the XmlNameSpaceManager class. This class is used to define namespaces at runtime.

Unfortunately SPWebConfigModification does not expose this class when defining the XPath expressions. If it did it would be possible to work with XML namespaces.

By the way, the team was able to work around the enterprise library configuration problem by moving the EntLib configuration settings into separate files.

Technorati Tags: SharePoint,Xml,configuration management

Creating a Feature to update the Web.Config

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 02 May 2008 03:56:00 +0000

I just wrapped up a feature that will update the web.config with custom settings. I did it by creating a feature receiver that loads web.config modifications from an xml configuration file and then applies them using the SPWebConfigModification class.

The entire process is straight forward. There are few subtle nuisances to SPWebConfigModification that make it tricky.

Creating New Web.Config Changes

Below is some sample code from my Feature. The ApplyWebConfigChange is responsible for creating an entry to the web.config.

private void ApplyWebConfigChange(SPWebApplication webApp, string ConfigurationChangeIdentifier, string ConfigurationName, string ConfigurationXPath, string ConfigurationValue)
{

    Collection&lt;SPWebConfigModification&gt; modsCollection = webApp.WebConfigModifications;

    SPWebConfigModification configMod = LookupModificationEntry(modsCollection, ConfigurationChangeIdentifier, ConfigurationName, ConfigurationXPath);
    if (configMod == null)
    {
        configMod = new SPWebConfigModification(ConfigurationName, ConfigurationXPath);
        configMod.Owner = ConfigurationChangeIdentifier;
        configMod.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
        modsCollection.Add(configMod);
    }

    configMod.Value = ConfigurationValue;
    webApp.Farm.Services.GetValue&lt;SPWebService&gt;().ApplyWebConfigModifications();
    webApp.Update();
}

The first step is to reference the collection that contains a list of SPWebConfigModification objects. This collection is accessed via the SPWebApplication.WebConfigModifications property.

The next step is to see if the change already exists in the WebConfigModifications collections. If it does then all I want to do is update it. If it is not there then I want to create it.

When creating new entries it is extremely important to make sure the SPWebConfigModification.Name is unique within the SPWebConfigModification.Path. If the name is not unique within the context of the path then you could end up with duplicate entries and you will not be able to remove the entry with SPWebConfigModification.

Tip: Get very comfortable with XPath since both the Path property and Name property have to be valid XPath syntax.

Make sure the Owner property is set to a unique value for your change. Later I will provide a quick and easy way to remove changes from SPWebConfigModification collection. My technique assumes that the Owner is unique for each modification.

Tip: Always set Type property to SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode. The two other options "EnsureAttribute" and "EnsureSection" will permanently change the web.config (as in you can never remove the items, even if you manually remove them they will return)

The last step is to call the SPWebService.ApplyWebConfigModifications and SPWebApplication.Update. Since my web application is deployed across multiple web front ends I need to call the ApplyWebConfigModifications. This routine will make sure the web.config change is made on all web front ends. If I only had one web front end then I could have just called SPWebApplication.Update.

Removing Web.Config Changes

Removing entries is really simple if you can assume that the Owner property is unique per change. In the routine below you can see that I will go through each item in the SPWebConfigModification. When I find an entry with the Owner set to a certain value I will remove it.

private void RemoveWebConfigChange(SPWebApplication webApp, string ConfigurationChangeIdentifier)
{
   
    Collection<SPWebConfigModification> modsCollection = webApp.WebConfigModifications;

    SPWebConfigModification deleteModification = null;

    foreach (SPWebConfigModification mod in modsCollection)
    {
        if (mod.Owner == ConfigurationChangeIdentifier)
        {
            deleteModification = mod;
            break;
        }
    }

    if (deleteModification != null)
    {
        modsCollection.Remove(deleteModification);
        webApp.Farm.Services.GetValue&lt;SPWebService&gt;().ApplyWebConfigModifications();
        webApp.Update();
    }
}

Here is an article that covers some more problems with SPWebConfigModification in more detail.

I learned a lot of great information about SPWebConfigModification here.

Technorati Tags: SharePoint,Configuration Management,Development

Instrumentation & Logging for SharePoint (Part 1)

jefferydalton@hotmail.com (Jeff Dalton) — Wed, 30 Apr 2008 01:24:00 +0000

Recently I have been defining how the development team will implement Instrumentation and Logging for SharePoint. I looked at a lot of different options and settled on the built-in .Net Framework diagnostic routines.

Logging

For logging I decided to use the System.Diagnostics.TraceSource class. This class was introduced with .Net Framework 2 and is a marked improvement over the original System.Diagnostics.Trace (my opinion).

Out-of-Box System.Diagnostics has trace listeners for Event Log, Console and File System. For grins I wrote a listener that will write to the SharePoint log system. It was very easy since the WSS SDK contains an example.

Three really nice features I like about the improved Diagnostics tracing classes are: Trace Options, Source Levels and Trace Filters

Trace Options provide a nice flexible way of indicating what data should be written along with the log message. Some examples include: Call Stack, Date & Time stamps, Thread ID, Process ID and Logical Call Stack. These options can be configured in the configuration file so the application does not need to even know about them.

Source Levels provide a nice alternative to the old Trace Switch class. A source level determines what type of messages should be logged (i.e. Critical, Error, All, ...).

Trace Filters provide a way to tell the Listener what types of errors it should log. This is really nice because it means that I can create File System listener that will write out every messages passed on by the Trace Source. At the same time my Trace Source could have an Event Log listener that still only writes critical messages to the Event Log.

The really great thing is all of these classes can be configured in a configuration file.

Below is a sample configuration I used while working with the Trace classes.

<system.diagnostics>
    <trace autoflush="true" />
    <sources>
        <source name="MyLogSource" switchValue="All">
            <listeners>
                <add name="consoleListener" />
                <add name="fileListener" />
                <add name="eventLogListener" />
                <add name="sharepointLogListener" />
            </listeners>
        </source>
    </sources>        
    <sharedListeners>
        <add name="consoleListener" type="System.Diagnostics.ConsoleTraceListener" />
        <add name="fileListener" initializeData="c:\temp\VolvoCWP.txt" type="System.Diagnostics.TextWriterTraceListener" traceOutputOptions="DateTime,ProcessId,LogicalOperationStack">
            <filter type="System.Diagnostics.EventTypeFilter" initializeData="All"/>
        </add>
        <add name="eventLogListener" type="System.Diagnostics.EventLogTraceListener" traceOutputOptions="DateTime,ProcessId,LogicalOperationStack" initializeData="JD Application">
            <filter type="System.Diagnostics.EventTypeFilter" initializeData="Critical"/>
        </add>
        <add name="sharepointLogListener" type="MyLogListener.SharePointTraceListener, MyLogListener, Version=1.0.0.0, Culture=neutral, PublicKeyToken=6bb1c46a01dcf0a2" ApplicationName="JeffD" ProductName="JeffD" CategoryName="Runtime">                
        </add>
    </sharedListeners>        
</system.diagnostics>

This configuration contains one Trace Source (MyLogSource). It is configured to log all messages that are logged with it (switchValue="All"). It has four listeners it will use to log the message (yes the message can get logged into four places... not very practical in the real world). The consoleListener, fileListener and sharepointLogListener are configured with filters that tell them to log every message. The eventLogListener is configured to only log messages marked as Critical.

The SharePointTraceListner was actually very simple to write. I was able to find a really great example of writing to the SharePoint log inside the WSS SDK. All I did was take that example and plug it into a standard Trace Listener.

Technorati Tags: SharePoint,Logging,Development,.Net Framework

Test Driven Development with SharePoint

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 25 Apr 2008 01:13:00 +0000

Lately I have been doing quite a bit of development inside of SharePoint. I am a big fan of Test Driven Development so naturally I wanted to carry over one of my favorite development techniques to SharePoint.

I wanted to share some of the obstacles I ran into along the way and techniques I used to overcome them.

Before I go on I want to point out that I am using Visual Studio 2008 for my development environment.

Missing Method Exception

One of the first problems I encountered was a Missing Method Exception when I would run the unit tests. I never did really find out what was happening. I did find a nice technique for working around it.

I modified the PostBuild routine of my Unit Tests to install my target DLL in the GAC. I also executed my tests in Debug mode. Doing both things overcame the issues. You can read the details here.

Testing Routines that require HTTPRuntime context

The next big challenge for me was to find a way to test routines that needed the HTTPRuntime, HTTPContext or SPContext objects.

I'm not a big fan of using mock objects when unit testing. And the thought of mocking any of those classes (or any of the SharePoint classes) turns my stomach.

For the most part my routines are built so they do not require runtime context. If I had to guess I would say < 5% of my tests are setup this way. Even though it is such a small part of my test footprint it is still a very important part.

Since I am using Visual Studio 2008 (this was available in VS.Net 2005 too) I am able to decorate those tests that require context with the HostType and UrlToTest attributes. Using these attributes will tell MSUnit to run my tests within the context of the IIS runtime.

[TestMethod()]
[HostType("ASP.Net")]
[UrlToTest("Url to SharePoint Page"]
public void FooBarTest
{
    string expected = "raBooF";
    string actual = MySharePointRoutine.RoutineThatUsesSPContext();
    Assert.AreEqual(expected, actual);
}

This works great, but there are some drawbacks.

1. Requires SharePoint site to be configured with Full trust

For MSUnit to hook into the IIS runtime context it has to do some tricky stuff. Unfortunately it doesn't work unless you configure the SharePoint site Full Trust.

One could argue that this invalidates my Unit Tests since it is quite possible that I will have some code that executes one way under Full Trust and another under WSS_Minimal Trust. While this is true I believe the chance of this happening is small.

2. Does not work with Publishing Pages

If you try to use a publishing page in the UrlToTest attribute the test will fail. Below is the error message.

The Web request 'SharePoint URL' completed successfully without running the test. This can occur when configuring the Web application for testing fails (an ASP.NET server error occurs when processing the request), or when no ASP.NET page is executed (the URL may point to an HTML page, a Web service, or a directory listing). Running tests in ASP.NET requires the URL to resolve to an ASP.NET page and for the page to execute properly up to the Load event. The response from the request is stored in the file 'WebRequestResponse_RelatedLinkPathTest_.html' with the test results; typically this file can be opened with a Web browser to view its contents.

I do not know exactly what this means, but I believe the problem may be caused by the fact that Publishing pages are virtual pages that map directly to a list item (i.e. they are not actually an ASPX page).

This kind of stinks because some of the tests may expect the SPContext.Current.ListItem property to be loaded with a valid list item. As a work around I discovered that I could could use the URL that will display list properties for a publishing page (Pages/Forms/DispForm.aspx?ID={List ID}).

To find create the DispForm Url I will first open a browser and point to /pages/forms/allitems.aspx (where pages represents the Publishing pages list). This will provide a list of all of the pages. I then choose the item I want and select View Properties.

Then I copy the URL from the web browser and I have what I need.

3. Difficult for Team Environments

Another problem is the fact that it is not possible (or I could never find a way) to parameterize the UrlToTest attribute. This means that the URL must be generalized so it works on all of the developer and build machines.

Microsoft has a great article that discusses issues teams will encounter when testing web pages (here). The article discuss's using the AspNetDeveloperServerHost attribute in the MSUnit. This attribute supports context parameters so tests can be made to work across teams. Unfortunately I was not able to AspNetDevelpoerServerHost to work with SharePoint Url's. When you think about all of the infrastructure needed to make WSS work it is not surprising that this does not work.

Conclusions

Test Driven Development with SharePoint is doable. The difficulties of setting up Unit Tests will vary depending on what your tests need to do.

Technorati Tags: SharePoint,Test Driven Development,Development,Visual Studio 2008

Creating a Rendering Template that supports Edit and Display modes

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 18 Apr 2008 02:57:00 +0000

I have been working on a Field Control rendering template that works for both Edit and Display modes. Along the way I learned quite a few things.

First I am not covering the steps to create a SharePoint field control. You can find a walk-through for creating a custom field control here.

DefaultTemplateName is not used in Display mode.

The first problem I ran into was the rendering of my control in Display mode. I had put in an override for the DefaultTemplateName property in my control. The documentation was vague but seemed to imply that the Display rendering template would be pulled from this value.

Well that is not the case. To get my custom rendering template loaded in Display mode I had to override the DisplayTemplateName property.

Once I had both DefaultTemplateName and DisplayTemplateName overridden I was in full control of the display.

Mixing the TemplateContainer and EditModePanel objects

With my rendering template I thought it would be best to just create one rendering template for both Edit and Display modes. The main reason I did this was so I could encapsulate all of the markup for a control in one rendering template.

My rendering template was nothing more than a ASP.Net User Control (ASCX file). By using a user control I had a nice container to edit the HTML mark-up for my control.

To give myself different looks for Edit and Display modes I decided to use the Publishing EditModePanel control. EditModePanels are a nice way of isolating a set of mark-up based on the page mode.

I put my EditModePanel controls inside my RenderingTemplate control. This is very important because it caused me grief later on.

<SharePoint:RenderingTemplate ID="VolvoCWPRelatedLinkControl" runat="server">
    <Template>    
        <SharePointPublishing:EditModePanel ID="RelatedLinkEditPanel" runat="server" PageDisplayMode="Edit" SuppressTag="true">

In the CreateChildControls routine of my control I needed to instantiate classes that were mapped to the controls in my ASCX file. According to the walk-through sample I should use the TemplateContainer.FindControl routine. Well this did not work because my controls were sitting inside an Edit Mode Panel.

To make this work I had to first find the Edit Mode Panel and then use the FindControl routine in its class.

this.empPanel = (EditModePanel)Utility.FindAndValidateControl(TemplateContainer, "RelatedLinkDisplayPanel");
this.empPanel.PreRender += new EventHandler(RelatedLinkDisplayPanel_PreRender);

this.lblTitle = (Label)Utility.FindAndValidateControl(this.empPanel, "tbRelatedLinkTitle");

this.repLinks = (Repeater)Utility.FindAndValidateControl(this.empPanel, "repRelatedLinkItems");

The FindAndValidateControl is a routine I wrote to do call the FindControl routine and make sure that the value passed back is not null.

public static object FindAndValidateControl(System.Web.UI.Control controlContainer, string controlName)
{            
    object returnValue = controlContainer.FindControl(controlName);
    if (returnValue == null)
        throw new ConfigurationErrorsException(string.Format("{0} not found. Corrupt control template.", controlName));

    return returnValue;
}

Once I figured out that the parent container mattered I was on my way. But it took me a few times staring at the "Control not found error" before I figured out what was going on. Just in case you are wondering I have not done a lot of ASP.Net control development so I realize I may have made a rookie mistake.

ItemFieldValue vs. Value

The next problem I ran into was a difference in the way the BaseFieldControl Value and ItemFieldValue properties work.

In my control I had put an override in for the Value property (per the Custom Control Walk-through). This works great when you are in Edit mode (I assume it also works when in New mode). But when in Display mode I discovered that the Value property was not being set. The documentation clearly states that when the control is loaded the Value is supposed to be given the same value as the ItemFieldValue property. Well this did not happen in my control.

To get around the issue I created a special handler for my Display Mode panel pre-render event.

public void RelatedLinkDisplayPanel_PreRender(object item, EventArgs e)
{
    EnsureChildControls();

    RelatedLinkValue fieldValue = new RelatedLinkValue(ItemFieldValue.ToString());
    QueryService queryService = new QueryService();
    DataTable queryResults = queryService.GetCategoryItems(fieldValue.Category, PropertyService.RelatedLinkPath, "PublishingPageContent");
    
    this.lblTitle.Text = fieldValue.Title;
    this.repLinks.DataSource =  queryResults;
    this.repLinks.DataBind();

}

In the PreRender event I call the EnsureChildControls (for the same reason why it should be called in the Value override) and I use the ItemFieldValue property.

Conclusion

I was able to consolidate my Edit and Display mode rendering templates into one control without too much effort. I did run into some gotchas, but nothing that made me think I had made a big mistake.

Technorati Tags: SharePoint,Development,Rendering Template,Field Control

Publishing Pages List Template ID

jefferydalton@hotmail.com (Jeff Dalton) — Wed, 16 Apr 2008 03:45:00 +0000

The Publishing Pages List BaseTemplate ID is 850.

It is not listed in the SPListTemplateType enumeration.

SharePoint Community Kit Wiki (or Lipstick on a Pig)

jefferydalton@hotmail.com (Jeff Dalton) — Mon, 14 Apr 2008 17:18:00 +0000

Recently the development team at work started using a Wiki to keep track of information that is important to the team. Since our company has standardized on using SharePoint for Collaboration solutions we used the Wiki featured delivered from Microsoft (via the Community Kit for SharePoint).

I must say I am very unimpressed by the SharePoint Wiki. I don't want to take anything away from the community effort that developed the Wiki template for SharePoint. I'm sure the community has done the best they can to develop a Wiki application on top of SharePoint (although it does appear to be missing some basic functionality one would expect from a Wiki.. Categories, Page Discussions). Perhaps SharePoint is not the best platform to host a Wiki.

Here is a short list of my main concerns with using the SharePoint Wiki.

1. Content is stored in a single list

As most know SharePoint starts to perform badly whenever a list contains more than 1000 items (this number is debated, what is not debated is that SharePoint performance degrades as list size grows).

The Wiki appears to be storing all pages directly in the root of a Document list. So after my Wiki has 1000 pages I will start to see a major slowdown. I am really surprised that categories (acting as Folders) were not implemented. To me this would have been a natural way to help reduce the impact of having a large list of Wiki pages.

2. Poor Search

This is my #1 complaint with our new Wiki. In my opinion a Wiki is only as good as its search engine. And unfortunately SharePoint search is still not as good as it can be.

Conclusion

When I compare the SharePoint Wiki features and functionality to something like ScrewTurn I am left with one thought.. the SharePoint Wiki sucks.

Technorati Tags: ShrarePoint,Community Kit for SharePoint,Wiki,ScrewTurn

FieldID class

jefferydalton@hotmail.com (Jeff Dalton) — Sun, 13 Apr 2008 18:31:00 +0000

I came across a very useful class in the Publishing library.

Microsoft.SharePoint.Publishing.FieldId

The FieldId class contains a list of commonly used field ids. I like it because it gives me an easy way to reference fields without having to worry about Internal Name vs. Display Name.

Since this is in the Publishing library you will have to have SharePoint 2007 to take advantage.

Technorati Tags: SharePoint,Field,Publishing

Missing Method Exception (or why my Unit Tests would not work with my SharePoint project)

jefferydalton@hotmail.com (Jeff Dalton) — Sun, 13 Apr 2008 18:30:00 +0000

I'm using some test driven development concepts on a little SharePoint development project.

I noticed that I kept running into a really strange error every time I added a new Method to my SharePoint project.

Whenever I would run my new Unit Test I would get a System.MissingMethodException error saying that the method in my SharePoint project could not be found.

Turns out that the problem was caused by the fact that the Unit Test project was loading my SharePoint assembly from the GAC. So for me to get the tests to work I had to install the latest version of my SharePoint assembly into the GAC and then close/open Visual Studio.

I like to take very small steps when I develop (add some functionality, test it, add some functionality, test it). So the idea of constantly updating the GAC and closing Visual Studio was not appealing.

I considered removing my SharePoint assembly from the GAC, but I put it in there for some legitimate reasons.

After much trial and error I found a solution that works okay (but is not ideal). I configured my Unit Test project to install the SharePoint dll in the GAC whenever the project is compiled. And I run the Unit Tests in Debug mode. With this setup I am able to get instant feedback from my changes.

As I said it is not ideal, but works.

To install my SharePoint dll in the GAC I added a Post Build event to my Unit Test project. It just runs the gacutil command.

To run my Unit Tests in debug mode I launch my tests using the Visual Studio Debug option rather than Run option.

I am using Visual Studio 2008 Test project for my Unit Tests.

Technorati Tags: SharePoint,Development,TDD,UnitTests,Visual Studio 2008

Six hours later...

jefferydalton@hotmail.com (Jeff Dalton) — Sat, 12 Apr 2008 02:55:00 +0000

I finally figured out why my Feature would not create my Site Columns and Content Types.

I mistyped the ElementManifests statement in the Feature definition.

Instead of using the ElementManifest tag I used the ElementFile tag. Uggg...

<Feature xmlns="http://schemas.microsoft.com/sharepoint/" ...>
    <ElementManifests>
        <ElementManifest Location="SiteColumns.xml"/> <!-- Right -->
        <ElementFile Location="SiteColumns.xml"/> <!-- Wrong -->
    </ElementManifests>
</Feature>

Rookie mistake. I didn't look closely at my Feature file because I was sure I had screwed up something in the element file that contained by Field definitions.

Technorati Tags: SharePoint,Features,Mistakes

Load Testing SharePoint (Lessons Learned) (Part 2)

jefferydalton@hotmail.com (Jeff Dalton) — Tue, 08 Apr 2008 01:45:00 +0000

In part 1 I discussed the setup of my load testing environment and tests. In part 2 I want to focus on the tests runs and what configuration I had to make to get them to work.

Bottlenecks oh Bottlenecks, wherefore art thou Bottlenecks

Right out of the gate I ran into trouble. My tests showed a bottleneck at 4 requests per second. The CPU was running @ 25 - 35% no matter what user load VSTS used (remember I was using goal based testing that would keep loading users until the goal was met). I tried several different tests and they all had the same result. So I knew there was a bottleneck somewhere.

I started by looking at the network. Specifically I focused on the virtual's network adaptor. I was worried that there was some sort of VMWare configuration problem. To test the network I used a file copy test (I uploaded and downloaded a large file to the web server). The test showed that the network was working just fine.

Then it hit me, the web server was configured to only support Integrated Windows authentication (NTLM). So I configured SharePoint (and IIS) to support Anonymous authentication. Bam, the bottleneck was gone.

Size Matters

So I ran my first set of tests. Unfortunately I noted a fairly significant difference between the out of the box Article page and our custom page.

Next I used the SharePoint Test Data Population Tool to create a site collection that contained 22,000 sites and about 50,000 pages.

Then I ran the tests again with some very surprising results. The out of the box Article page when from 114 requests per second to 56 requests per second. That's right we recorded an almost 100% decrease in performance just due to the size of the site collection.

Our custom page's did not experience the same percentage of slow down (actually their performance improved, but that was due to improvements we made to the code).

Summing Up

Performance testing SharePoint is a tedious but necessary task. Here are my lessons learned from the exercise.

1. Plan, Plan, Plan

Establish the test goals

Determine which tests you need to run

Determine which tools you will use

Determine the setup/configuration of your SharePoint and Load test environments.

2. Dry run the tests (leave plenty of time to work through issues)

3. Don't test with an empty site collection

Technorati Tags: SharePoint,Performance,visual Studio Team System,Load Testing

Musing on CrossListQueryCache class

jefferydalton@hotmail.com (Jeff Dalton) — Mon, 07 Apr 2008 02:51:00 +0000

Summary

The SharePoint CrossListQueryCache class provides a highly scalable way to run site wide queries. It provides all the power of SPSiteDataQuery without the performance implications. However it has some major gotchas that you should be aware of before using.

I'm always looking for ways to improve application performance (and with SharePoint I have more opportunities than time). I was focused on some code we use to pull list items. This code is in the critical path for our runtime performance and scalability. It is the perfect place for cached data query. So I started looking at different ways to do data querying in SharePoint.

My first stop was on Google where I found some information about the SharePoint SPSiteDataQuery object. According to Eric Shupps (The SharePoint Cowboy :D) SPSiteDataQuery is great for small data sets and small to medium traffic environments. Eric actually recommends using the good ole GetListItems method. Well we are using GetListItems today, so I was not encouraged. Waldek Mastykarz put together a really nice posting that compares several different approaches for querying lists (just what I needed).

Even though I had read the warnings about SPSiteDataQuery I decided to check it out (for some reason I really like to repeat the mistakes of others ;-). I really like the functionality provided by the class. I used Reflector to look under the covers. I didn't seen anything crazy going on (makes a call to COM method called CrossListQuery... I didn't bother chasing it to the SQL layer). I ran some performance tests with it, but didn't see a compelling reason to dump GetListItems for this routine.

Next I jumped into the CrossListQueryInfo/CrossListQueryCache routines. It is worth mentioning that these are in the Microsoft.SharePoint.Publishing library which means you have to have SharePoint 2007 (WSS v3 does not include this library). What I found shocked and delighted me (hmm, doesn't sound quite right).

First the shock...

According to the documentation CrossListQueryCache was built specifically for what I needed to do. Query one or more lists and store the results in cache so subsequent calls are at near lightening speed.

But when I ran my tests I was seeing no better performance than SPSiteDataQuery (all you haters saying CrossListQueryInfo.UseCache must be set to true, bite your tongue and keep reading). I assumed I was doing something wrong, so I cracked open the worlds best documents (the source code).

I used Reflector (one of the best dam tools ever developed) to look at the routine I was using, CrossListQueryCache.GetSiteData(SPWeb). Well according to the code there is no caching going on, infact this routine is nothing more than a wrapper for SPSiteDataQuery. So a little more checking and I discovered that something strange was afoot in the CrossListQueryCache library.

There are 4 overloads for CrossListQueryCache.GetSiteData. Two of the overloads cache the results and two do not.

CrossListQueryCache Routine	Uses Cache
GetSiteData (SPSite)	Yes
GetSiteData (SPWeb)	No
GetSiteData (SPSite, String)	Yes
GetSiteData (SPWeb, SPSiteDataQuery)	No

So unless you are using one of the overloads that supports caching (and have the CrossListQueryInfo.UseCache property set to true) you might as well use SPSiteDataQuery.

There is one other gotcha with the overloads that support cache. Your query cannot contain "<UserId/>" tag. If it does then cache is not used.

Next the delight...

When you use the GetSiteData routines that support caching you see major performance improvements. In fact my testing showed that the performance of CrossListQueryCache was as good as the Content By Query Control (not surprising since they are both caching their results).

Code Snippet

CrossListQueryInfo clqi = new CrossListQueryInfo();
clqi.Query = "<OrderBy><FieldRef Ascending=\"FALSE\" Name=\"Title\" /></OrderBy><Where><BeginsWith><FieldRef Name=\"Title\" /><Value Type=\"Text\">Test</Value></BeginsWith></Where>";
clqi.ViewFields = "<FieldRef Name=\"Title\" /><FieldRef Name=\"ContentType\" />";
clqi.Lists = "<Lists BaseType=\"1\" />";
clqi.Webs = "<Webs Scope=\"SiteCollection\" />";
clqi.UseCache = true;

CrossListQueryCache clqc = new CrossListQueryCache(clqi);
DataTable tbl = clqc.GetSiteData(SPContext.Current.Web);

The code above (which does not use caching since it passes in the Web) averaged 0.0500 seconds.

That same code slightly modified (see below) averaged 0.0008 seconds.

Code Snippet

CrossListQueryInfo clqi = new CrossListQueryInfo();
clqi.Query = "<OrderBy><FieldRef Ascending=\"FALSE\" Name=\"Title\" /></OrderBy><Where><BeginsWith><FieldRef Name=\"Title\" /><Value Type=\"Text\">Test</Value></BeginsWith></Where>";
clqi.ViewFields = "<FieldRef Name=\"Title\" /><FieldRef Name=\"ContentType\" />";
clqi.Lists = "<Lists BaseType=\"1\" />";
clqi.Webs = "<Webs Scope=\"SiteCollection\" />";
clqi.UseCache = true;

CrossListQueryCache clqc = new CrossListQueryCache(clqi);
DataTable tbl = clqc.GetSiteData(SPContext.Current.Site, CrossListQueryCache.ContextUrl());

Slightly Different Results

As I said the performance is great, but one thing really bothered me. The DataTable results did not match. Everything about the code is identical (same query, same CrossListQueryInfo configuration), but different results.

I discovered that the routines that actually use cache contain some extra code that does a check to see if the item is checked-in. If it is then it shows it, if not then it does not. While the code that does not use cache returns all items that match the query (even those that are not checked-in).

I was worried that the routines were not using the same technique to query across the site lists, so I did some digging with Reflector. What I found is that they both use the SPWeb.GetSiteData to execute the query. The routines that cache the results do a little extra work to trim the results. Just take a look at CachedArea.GetCrossListQuery if you want to see what is happening.

Anyway the end result was that I found that if used correctly the CrossListQueryCache object operates with the same efficiency of the Content Query Control. It provides an out of the box way to have your GetSiteData results cached. I highly recommended it if you are using SharePoint 2007 and not WSS 3.

Technorati Tags: SharePoint,Performance,Development

Load Testing SharePoint (Lessons Learned) (Part 1)

jefferydalton@hotmail.com (Jeff Dalton) — Sun, 16 Mar 2008 18:26:00 +0000

I want to share some experiences from a recent load testing exercise I performed with SharePoint.

I am working on a web content management project that requires A LOT of customizations to SharePoint. The team needed to understand how these customizations would perform and scale. So I dusted off my limited knowledge of load testing with Visual Studio Team Test and went to work.

First I established my testing environment. I started by setting up my test agent. I was able to commandeer an old R&D server (HP 385, Dual Pentium III, 2 GB RAM, 1 GB NIC) for my cause. I wasn't sure, but suspected that this would be more than enough machine for my needs (oh yeah, its business time). Setup was a breeze, I just installed Visual Studio Team Test 2008 and my load agent was ready.

Next I needed to prepare my SharePoint environment. I already had a virtualized (is that a word) environment for SharePoint. Actually I had two to choose from, one dedicated to integration testing for the development team and another for user acceptance testing. I decided to go with the integration test environment because it was located inside of the corporate network (no messy firewalls).

Next I needed to create my web tests. This step required me to put on a thinking cap. There are many different approaches for load testing and I am not going to discuss them in this article. I chose to create tests that would isolate specific pieces of functionality (rather than trying to mimic usage patterns). By using this approach I could clearly understand how our customizations would compare to out of the box SharePoint.

Here is a list of my tests

ID	Test	Description
1	HTML page with Hello World text	This is not a SharePoint item. I created it so I could have an overall for the test environment. The results from this test represent the fastest the server can go. This will act as a high water mark for me to compare my results to.
2	SharePoint Publishing Article Page	This is an out of the box SharePoint item. The results from this test represent a baseline for me to compare my test results against. My tests results should be in the same neighborhood as these test results.
3	Our Customized SharePoint Page Layout with one customized control.	This represents a very lightweight version of our customizations.
4	Our Customized SharePoint Page Layout with two customized controls.	The idea is to add a small piece of functionality so I can understand how the change impacts the test results.

I ran each one of those tests independently. The idea was to take a single test and stress the system.

I used goal based load tests in VSTT (a new feature for VS.Net 2008). My goal was to stress the Web Server so the CPU would run between 70 - 90%. Goal based testing is why I decided to avoid the firewalls (Goal based testing requires that the load agent communicate with the performance counters on the target machines).

VSTT will continue to load transactions until the goal is reached. If the goal is never reached then VSTT will load to the maximum number of requests/users specified in the load test.

Now I had everything ready to go... or so I thought. In Part 2 I will go into detail about tweaks I had to make to my SharePoint environment so I could get the load tests to work correctly.

SPTD (no its not a venereal disease, but can cause hives)

jefferydalton@hotmail.com (Jeff Dalton) — Sat, 08 Mar 2008 19:41:00 +0000

Test Data Load Tool is a nifty little utility for creating data in your WSS or SharePoint site. You can download the tool at CodePlex http://www.codeplex.com/sptdatapop.

The tool provides a repeatable way to create test data for your SharePoint sites. It is extremely cool and even with the warts it is worth the time to sit and learn it. If you are a tester for SharePoint solutions you need to add this one to your bag of tricks. If you are a developer you will appreciate the base framework and potential.

The tool comes in two flavors (WSSDW.exe) and (MOSSDW.exe). They are in separate download packages so if you want to load data for things like SSO, publishing portals, audiences (SharePoint features) you will need to get both downloads. The tools are fairly easy to grasp, just take some time and go through the help files and samples.

That said the tools are buggy. Here are some things I learned the hard way.

The Delete option works about half the time... I finally quite using it and went another route. I'm sure the code is working as intended, but when you say delete you expect everything to go. Perhaps a -force option is needed with the delete.

ModifyExisting depends on "ModifyOne" interface being implemented, if the interface is not implemented no updates are made. It is extremely frustrating when you want to do something simple (ex. change the title of a web) and discover that the Webs class does not do anything in ModifyOne.

Pages have to be checked out to modify them (this may make you go Duh, but I was hoping to not have to worry about things like checkin/checkout/publish since it is a test tool). I ended up having 3 statements per page (one with OpenorCreate, one with checkout, one with publish).

Don't expect your script to work the first (or second, or third) time. Plan on having plenty of time to work through issues as you learn about the subtle nuances of the tool. It took me a good 2 days to get the kinks worked out of my script, but once I had it done I could bounce from environment to environment and setup a site that was test worthy (Seinfeld reference).

Depending on the amount / type of work your script needs to do it can take a while to run the script. I uploaded about 200 images and documents. The first run took about 30 minutes to do the upload, but every run after that was < 1 minute.

Zune Wiener

jefferydalton@hotmail.com (Jeff Dalton) — Sat, 08 Mar 2008 01:49:00 +0000

I was one of the lucky Zune winners at SPC2008. Yeah! I never win jack so I'm in shock.

SPC 2008 Day 4

jefferydalton@hotmail.com (Jeff Dalton) — Fri, 07 Mar 2008 01:07:00 +0000

This was the final day of SPC 2008. It was a great conference and I'm really glad I came.

The day started with a session from Russ White in PSS. He was talking about how MSFT PSS troubleshoots SharePoint problems. The title was a little misleading since Russ spent about half of the talk providing best practices for capturing information about the problem (nothing to do with SharePoint). The actual SharePoint data was a little lite. Would have been nice to focus on SharePoint.

Next I went to Tyler Butlers presentation on deploying solutions and content throughout the farm. Tyler gave some solid practical advice for the content deployment tools. He also showed some information about known problems with the tools. He mentioned that in the next 90 days an update that addresses several more fixes should be coming out.

The final session was presented by Andy Hopkins and Paul Learning. They talked about their findings from creating a highly scalable SharePoint solution. The information they shared was priceless. I'm so happy that MSFT made the investment to really find out what SharePoint can do. This made me feel much better about what we are doing.