Chris O'Brien's blog

Renewed as SharePoint MVP

noreply@blogger.com (Chris O'Brien) — Sun, 05 Jul 2009 21:08:00 +0000

Like several others, I heard the good news from Microsoft last week that I’m a SharePoint MVP for another year. I’m really pleased to get the award again, as it’s nice to be recognized by Microsoft for what you’ve done. I find it useful (to me anyway!) to have a bit of reflection on what I’ve been up to for the last year. The big things were probably:

A big re-write of my popular SharePoint Content Deployment Wizard tool, to support added functionality such as scripted deployments
Release of the SharePoint Config Store framework, for storing configuration values in SharePoint development
Release of the sister project, the SharePoint Language Store framework, for helping to build multi-lingual SharePoint sites
Speaking at the European SharePoint Best Practices Conference on deployment approaches/best practices for SharePoint sites
Being on various expert panels at different events (such as the Best Practices Conference)
Other speaking gigs, like my Lessons learnt/top 5 tips for building WCM sites in SharePoint session to a record-breaking audience at the UK SharePoint user group
31 new articles and answers to over 100 questions/comments on this blog

A busy year to be sure. I just want to say thanks to everybody that went to the trouble of passing on their thanks or feedback on some of this stuff – I really appreciate it..

My checklist for optimizing SharePoint sites

noreply@blogger.com (Chris O'Brien) — Sun, 21 Jun 2009 22:47:00 +0000

Optimization is probably one of my favorite SharePoint topics – it seems there’s always a new trick to learn, and I love the fact that a minor tweak can have a dramatic impact on how well your site works. There are some great resources out there now for information on how to boost performance, but it strikes me that there isn’t one single paper or article which contains all the aspects I personally consider or have been relevant to me on past projects. Hence this article is essentially an aggregation of some of these sources. The aim isn’t for it to be “complete” (if there is such a thing) or authoritative in any way, and I know I wouldn’t be the first person to blog on this subject – it really is just a reminder list for me to refer to, but if it helps you also that’s great.

Key resources:

How to Optimize a SharePoint Server 2007 Web Content Management Site for Performance
Optimizing Office SharePoint Server for WAN environments
Prescriptive Guidance for SharePoint Server 2007 Web Content Management Sites
Tune Web Server Performance (Office SharePoint Server)
Many blog articles/presentations by AC, Spence and others
Steve Souder’s (of Google) blog

Before I break into the list, remember that different optimization measures will have different effects depending on your circumstances - having the best checklist in the world is really no substitute for the SharePoint architect’s ability to identify what is likely to be the bottleneck for a given implementation. Additionally my list is quite “developer-focused”, but if the infrastructure aspects of the implementation is the limiting factor (e.g. latency between web servers and SQL, amount of usable RAM etc.) you can probably optimize code until you’re blue in the face without solving your performance problem.

My list is broken down into sections, and some of the items have links to some useful resource or other which you might find helpful if the topic is new to you.

Configuration

Items in this section typically have a big impact – in particular output caching, which is the first thing to consider on WCM sites for example.

Output caching (or declarative caching at user control level)
HTTP/IIS compression, including .aspx files
BLOB caching
Object caching configuration
Is networking configured correctly e.g. cards set to gigabit
Is search indexing configured appropriately e.g. consider dedicated WFE for indexing, crawl frequency/type
Move CSS/JS files out of Style Library (to another library or to filesystem) due to HTTP 304s/BLOB cache bug
Check using Fiddler/YSlow (more on this later) or similar for unexpected HTTP 304, 401, 404 codes

Code/development

Test with SPDisposeCheck to ensure no memory leaks
Measure page payload weight (again, YSlow useful here) – aim to reduce as far as possible
Eliminate unnecessary ViewState! A good dev technique may be to turn off ViewState in web.config and then override it in controls/pages which really do need it (haven’t tried this, but keep meaning to).
Ensure not downloading core.js etc for anonymous users/delay-loading for authenticated.
Ensure image sizes are small/images are optimized for web (still comes up!)
Always ensure height/width attributes specified on images
Ensure custom CSS is factored correctly
Don’t forget client-side code efficiency e.g. CSS, jQuery selectors
Consider using a code-profiling tool such as ANTS profiler or dotTrace to identify sub-optimal code
Ensure general good coding practice e.g. your optimization work may well be undone if your code is badly-written. Accidentally doing unnecessary processing in a loop (in server-side OR client-side code is one example I’ve seen many times)
Be wary of performance impact of custom HTTP modules – I frequently see huge perf gains when running load tests with such modules enabled/disabled. Removing core.js as a post-processing step within a HTTP module is NOT the right approach kids! (No really, I’ve seen it done..)

Code/development – advanced

These tips might only really be necessary if you’re implementing a high-traffic site, but are worth considering:

Consider factoring custom JS/CSS into fewer files – fewer HTTP requests is better
Consider “minifying” JS
Consider using CSS to cluster background images (all images stitched into one for fewer HTTP requests) – AC might have mentioned somewhere an online service to automate this..

Architecture

You should consider this section absolutely incomplete – hopefully there’s enough here to dissuade anybody of the notion that it’s only about code though! The first two in particular are key.

64-bit/sufficient available RAM etc.
Latency between web and SQL – MS suggest a ping response of less than 1 millisecond
Consider a CDN solution (e.g. Akamai, Limelight etc.) if users are geographically far away from servers (and y0ur client can afford it!)
Ensure not using web garden with BLOB cache (known BLOB cache issue)
Are app pool settings correct – check for excessive app pool recycling by enabling logging to the Windows event log using - cscript adsutil.vbs Set w3svc/AppPools/ <YourAppPoolName> /LogEventOnRecycle 255. Once or twice per day is probably the maximum you’re hoping for.
Is storage correctly architected e.g. are OS, data, logs on different disk spindles?
Is storage correctly configured (e.g. SAN configuration) – are your LUNS giving you sufficient IOPS?!

Using YSlow to help optimize websites

One particular tool worthy of discussion when we’re talking about optimizing websites is YSlow – if you haven’t come across it yet, this is a Firefox plugin which extends Firebug) developed by Yahoo! developers which uses their optimization ruleset (customizable) to report on and help further streamline your site. The tool focuses on communication between the server and browser – clearly a browser tool like this can’t identify any back-end issues, but it’s always interesting to run a site through the checks.

The first tab gives a ‘grade’ for your site based on the ruleset, helping you identify areas for improvement (I’ve highlighted where the page stops and the YSlow console starts with a red box):

The next tab provides a breakdown of files downloaded by the request, and allows me to see their size and headers etc. In particular I can see the expiry date and Etag and so on which the files are being tagged with so they can be cached locally:

The statistics tab provides some good analysis on the page weight and also illustrates the difference between a first visit and subsequent visits where appropriately tagged files will be served from the browser cache:

Finally the Net tab in Firebug is also interesting, as this shows me how files were downloaded (sequential or in parallel) – the most recent browsers do a much better job of this, with IE8 and FF3 being able to open 6 channels per URL domain to download files in parallel, but note that IE7 could only open 2.

From this, I also see the http://sharepoint.microsoft.com site (great site btw) I’ve been analyzing also displays the BLOB cache issue I discussed recently (now confirmed as a bug) where incorrect headers are added to files stored in the Style Library, causing lots of unnecessary HTTP 304s to go across the wire. So YSlow does give a great insight – however I do agree with Jeff Atwood’s reminder that some of the reported “issues” may not be the biggest concern for your site. As always, apply common sense.

Summary

Optimization is a deep topic, and this checklist is simply my reminder of some of the things to consider, though in reality a solid understanding of the nuts and bolts is required to really architect and develop high-performing sites. Tools like YSlow can also help with some aspects of optimization.

So which optimization nuggets did I miss which you consider? Feel free to leave a comment and add to this list..

More on optimization, HTTP 304s etc. – a solution?

noreply@blogger.com (Chris O'Brien) — Sun, 31 May 2009 21:06:00 +0000

In my last post Optimization, BLOB caching and HTTP 304s, I did a fairly lengthy walk-through on an issue I’d experienced with SharePoint publishing sites. A few people commented, mainly saying they’d noticed the same thing, but there have been further developments and findings I wanted to share!

Quick recap

Under certain circumstances some files in SharePoint are always re-requested by the browser despite being present in the browser cache (“Temporary internet files”). Specifically this is observed for files stored in the Style Library and Master Page Gallery, for anonymous users. Although SharePoint responds with a HTTP 304 to say the cached file can indeed be used (as opposed to sending the file itself again), we effectively have an unnecessary round-trip to the server for each file – and there could be many such files when all the page’s images/CSS/JS files are considered. This extra network traffic can have a tangible impact on site performance, and this is magnified if the user is geographically far away from the server.

A solution?

Waldek and I have been tossing a few development matters around recently over e-mail, and he was curious enough to investigate this issue for himself. After reproducing it and playing around for some time, Waldek discovered that flushing the disk-based cache seems to cause a change in behaviour – or in layman’s terms, fixes everything. To be more specific, we’re assuming it’s a flush of the BLOB cache which is having the affect – in both Waldek’s test and my subsequent validation, the object cache was also flushed as well:

After the OK button is hit on this page, the problem seems to go away completely, so now when the page is accessed now for the first time as an anonymous user, the correct ‘max-age’ header is added to the files (as per the BLOB cache declaration in web.config) – contrast the ‘max-age=86400’ header on the Style Library files with what I documented in my last post:

This means that on subsequent requests, the Style Library files are served directly from the browser cache with no 304 round-trip:

This is great news, as it means the issue I described is essentially a non-issue, and there is therefore no performance penalty for storing files in the publishing Style Library.

So what gives?

I’m now wondering if this is just a ‘gotcha’ with BLOB caching and publishing sites. I know other people have run into the original issue due to the comments on my previous post, and interestingly enough one poster said they use reverse proxy techniques specifically to deal with this issue. Could it really be that everybody who sees this behaviour just didn’t flush the BLOB cache somewhere along the way, when it’s actually a required step? Or is the testing that Waldek and I did flawed in some way? Or indeed, was my initial investigation flawed despite the fact others reported the same issue?

I am interested to hear from you on this – if you can reproduce the problem I’ve described with a publishing site you’ve developed, does flushing the BLOB cache solve it for you as described here? Leave a comment and let us know!

Good work Waldek :-)

Optimization, BLOB caching and HTTP 304s

noreply@blogger.com (Chris O'Brien) — Sun, 17 May 2009 00:21:00 +0000

There's been an interesting mini-debate going on recently in terms of where to store static assets used by your site - images, CSS, JS files and so on. Broadly the two approaches can be characterized as:

Developer-centric - store assets on the filesystem, perhaps in the 12 hive
Author-centric - store assets in the content database, perhaps in the Style Library which comes with publishing sites

Needless to say these options offer different pros and cons depending on your requirements - Servé Hermans offers a good analysis in To package or not to package: that is the question. However, I want to throw another point into the debate - performance, specifically for anonymous users. Frequently, this is an audience I care deeply about since some of the WCM sites I work on often have forecast ratios of 80% anonymous vs. 20% authenticated users. Recently I was asked to help optimize an under-performing airline site built on MOSS - as usual the problem was a combination of several things, but one of the high-impact items was this decision to store assets in one location over the other. In this post I'll explain what the effect on performance is and why you should consider this when building your site.

The problem

Once they've been loaded the first time, most of the static files a website uses should be served from the user's local browser cache ("Temporary internet files") - without this, the internet would be seriously slow. Consider how much slower a web page loads when you do a hard refresh (ctrl+F5) compared to normal - this is because all the images are forced to be re-downloaded rather than served from the browser cache. Unfortunately, for files stored in some common SharePoint libraries/galleries (i.e. the author-centric approach) SharePoint doesn't deal with this quite right in some scenarios - most of the gain is there, but despite having the image locally, the browser still makes a request for the image - the conversation goes like this (for EACH image on the page!):

Browser: I need this image please - I cached it last time I came at [date/time], but for all I know it's changed since then.
Server: No need dude, it's not changed so just use your local copy (in the form of a HTTP 304 - "Not modified")
Browser: Fair enough, cheers.

This essentially happens because the file was not served with a "cacheability" HTTP header to begin with. Needless to say, this adds significant time to the page load when you have 30+ images/CSS/JS files referenced on your page - potentially several seconds in my experience (under some circumstances), which of course is a huge deal. If say, the user is in Europe but the servers are in the U.S., then suddenly this kind of network chatter is something we need to address. Needless to say, in the majority of cases we're happy to cache these files for a period since they don't all change too often, and we get better performance as a result.

The Solution (for some SharePoint libraries *)

Mike Hodnick points us to part of the solution in his highly-recommended article Eliminating "304" status codes with SharePoint web folder resources. Essentially, SharePoint's BLOB caching feature saves the day since it serves the image with a "max-age" value on the HTTP header, meaning the browser knows it can use it's local copy of the file until this date. This only happens when BLOB caching is enabled and has the max-age attribute like this (here set to 84600 seconds = 24 hours):

<BlobCache location="C:\blobCache" path="\.(gif|jpg|png|css|js|aspx)$" maxSize="10" enabled="true" max-age="86400" />

When we configure the BLOB cache like this we are, in effect, specifying that it's OK to cache static files for a certain period, so the "cacheable" header gets added. HOWEVER, what Mike doesn't cover is that this only happens for authenticated users - files served out of common content DB locations such as the Style Library and Master Page Gallery still do not get served correctly to anonymous users. Note this isn't all SharePoint libraries though - so we need to be clear on exactly when this problem occurs.

* Scope of this problem/solution

Before drilling down any deeper, let's stop for a moment and consider the scope of what we're discussing - a site with:

Anonymous users
Files stored in some libraries - I'm not 100% sure of the pattern but discuss it later - the Style Library and Master Page Gallery are known culprits however. Other OOTB libraries such as SiteCollectionImages do not have the problem.

If you don't have this combination of circumstances, you likely don't have the problem. For those who do, we're now going to look closer at what's going on, before concluding with how we can work around the issue at the end.

Drilling deeper

For a site which does have the above combination of circumstances, we can see the issue with Fiddler - as an anonymous user browsing to page I've already visited, I see a stack of 304s meaning the browser is re-requesting all these files:

However, if I'm authenticated and I navigate to the same page, I only see the HTTP 200 for the actual page, no 304s:

Hence we can conclude it works fine for authenticated users but not for anonymous users.

So what can we do for our poor anonymous users (who might be in the majority) if we're storing files in the problematic libraries? Well, here's where I draw a blank unfortunately. Optimizing Office SharePoint Server for WAN environments on TechNet has this to say on the matter:

Some lists don't work by default for anonymous users. If there are anonymous users accessing the site, permissions need to be manually configured for the following lists in order to have items within them cached:

Master Page Gallery
Style Library

Aha! So we need to change some permissions - fine. This seems to indicate that it is, in fact, possible to get the correct cache headers added to files served from these locations. Unfortunately, I simply cannot find what permissions need to be changed, and nobody on the internet (including the TechNet article) seems to detail what. The only logical setting is the Anonymous Access options for the list - these are all clear by default, but adding the 'View Items' permission (as shown below) does not change anything:

As a sidenote, the setting above is (I believe) effectively granting read permissions to the identity which is used for anonymous access to the associated IIS site. So in IIS 7.0, I'm fairly sure you'd achieve the same thing by doing this:

So the problem does not go away when anonymous users are granted the 'View Items permission, and what I find interesting about this is that a closer look with Fiddler reveals some inconsistencies. The image below shows me browsing to a page anonymously for the first time, and to save you the hassle we can derive the following findings:

Files served from the 'SiteCollectionImages' library are given the correct max-age header (perhaps expected, since not one of the known 'problem libraries' e.g. Style Library)
Files served from the '_layouts' folder are given a different max-age header (expected, settings from the IIS site are used here)
Some files in the Style Library are in fact given a the correct max-age header! (not expected)

So the 2 questions which strike me here are:

Why are some files being served from 'Style Library' with the correct header when most aren't?
Why can SharePoint add the 'max-age' header to files in the 'SiteCollectionImages' library but not the 'Style Library'?

The first one is a mystery to me - it's perhaps not too important, but I can't work it out. The second one might be down to how the libraries are provisioned - the 'Style Library' is provisioned by declarative XML in the 'PublishingResources' Feature, whereas the 'SiteCollectionImages' library is provisioned in code using the same Feature's activation receiver. Could this be the key factor? I don't know, but I'd certainly be interested if anyone can put me straight - either on this or the mystery "permissions change" required to make BLOB caching deal with libraries such as the 'Style Library'.

Conclusion

The key takeaway here is that for sites which want to take advantage of the browser caching for static files (for performance reasons) and have anonymous users, we need to be careful where we put our images/CSS/JS files as per Mike Hodnick's general message. If we want to use the author-centric approach and store things in SharePoint libraries, we need to consider which libraries (and test) if we will have the 304 problem. Alternatively, we can choose to store these files on the filesystem (the developer-centric approach) and use a virtual directory with the appropriate cacheability settings to suit our needs. My suggestion would be to use a custom virtual directory for full control of this, since the default settings on the '_layouts' directory ("cache for 1 year") are unlikely to be appropriate.

Fix to my Language Store framework for multi-lingual sites

noreply@blogger.com (Chris O'Brien) — Mon, 04 May 2009 22:29:00 +0000

In my last post, I talked about a fix to my Config Store framework for an issue which manifested itself on certain SharePoint builds, with Windows 2008 and a recent cumulative update seeming to be the trigger. Some of you may know that I produced a sister project to this one called the 'Language Store', which is designed to help build multi-lingual SharePoint sites - since this framework is built off the same underlying XML and plumbing, this solution was also affected. So this post is just a short one to say that the fix has now been applied to the Language Store framework, and the new version is now available on Codeplex at http://splanguagestore.codeplex.com.

Problem/solution

The problem was effectively that items in the SharePoint list could no longer be edited - well, in fact they could be updated using code, but the list form .aspx pages were not showing the fields correctly so items couldn't be edited in the UI. Since it kind of defeats the point of SharePoint to have to write code to update list items (!), this was a big issue on affected builds. Interestingly some users reported working around the issue by removing/re-adding the content type from the list in the browser, but happily this is no longer necessary since the root issue has now been resolved. The problem was traced to some incorrect XML in my FieldRef elements - see the last post Fix to my Config Store framework and list provisioning tips for the full info.

General recap - the Language Store

If you're still reading, I figure some folks would welcome a reminder/intro on what the Language Store actually does - it's not about replacing SharePoint's variations functionality which is commonly used on multi-lingual sites. I noticed Spence gave it a better name in an e-mail recently where he described it as a 'term store' for multi-lingual sites - this actually captures what it does far better than my name for it. Effectively the idea is to provide a framework for the many small strings of text which are not part of authored page content which need to be translated and displayed in the appropriate language. As an example, here is a page from the BBC site where I've highlighted all the strings which may need to be translated but which don't belong to a particular page:

There are many of these in a typical multi-lingual site, and to help deal with this requirement the Language Store framework provides the following:

SharePoint list/content type/site columns etc.
API to retrieve items with a single line of code
Granular caching for high-performance
Packaged as a .wsp for simple deployment
All source code/XML freely available

If you want to find out more, see Building multi-lingual SharePoint sites - introducing the Language Store. The solution can be downloaded from the Codeplex site at http://splanguagestore.codeplex.com.

Apologies to existing users who were affected by the issue.

Fix to my Config Store framework and list provisioning tips

noreply@blogger.com (Chris O'Brien) — Wed, 22 Apr 2009 23:42:00 +0000

Had a couple of reports recently of an issue with my Config Store solution, which provides a framework for using a SharePoint list to store configuration values. If you're using the Config Store this article will definitely be of interest to you, but I've also picked up a couple of general tips on list provisioning which I want to pass on. I have to thank Richard Browne (no blog) of my old company cScape, as the fix and several of the tips have come from him - as well as alerting me to the problem, he also managed to fix it before I did, so many thanks and much kudos mate :-)

Config Store problem

Under some circumstances, fields in the Config Store list were not editable because they no longer appeared on the list edit form (EditForm.aspx). So instead of having 4 editable fields, only the 'Config name' field shows in the form:

I've not fully worked out the pattern, but I think the problem may only appear if you provision the list on a server which has the October or December Cumulative Update installed - either that or it's a difference between Windows 2003 and Windows 2008 environments (which would be even more bizarre). Either way, it seems something changed in the way the provisioning XML was handled somewhere. This is why the problem was undetected in the earlier releases.

I had seen this problem before - but only when the list was moved using Content Deployment (e.g. using the Content Deployment Wizard) - the original 'source' list was always fine. We managed to work around this by writing some code which 're-added' the fields to the list from the content type, since they were always actually present on the content type and the data was still corrected stored. Having to run this code every time we deployed the list was an irritation rather than critical, but something I wanted to get to the bottom of - however, on finding some folks were running into this in 'normal' use meant that it became a bigger issue.

The cause

I always knew the problem would be down to a mistake in the provisioning XML, but since I'd looked for it on previous occasions I knew it was something I was seeing but not seeing. In my case, Richard spotted that I was using the wrong value in my FieldRef elements under the ContentType element - I was mistakenly thinking that the 'Name' attribute needed to match up with the ''StaticName' attribute given to the field; the documentation says this attribute contains the internal name of the field. So my FieldRefs looked like this:

<ContentType ID="0x0100E3438B2389F84cc3965600BC16BF32E7" Name="Config item" 
               Group="Config Store content types" Description="Represents an item in the config store." Version="0">
    <FieldRefs>
      <FieldRef ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}" Name="ConfigCategory" Required="TRUE" />
      <FieldRef ID="{BD413479-48AB-41f5-8040-918F32EBBCC5}" Name="ConfigValue" Required="TRUE" />
      <FieldRef ID="{84D42C64-D0BD-4c76-8ED3-0A9E0D261111}" Name="ConfigItemDescription" />
    </FieldRefs>
  </ContentType>

..to match up with fields which looked like this:

<Field ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}"
      Name="Config category"
      DisplayName="Config category"
      StaticName="ConfigCategory"
      ....
      ....
/>

The CORRECTED version looks like this (note the change in value for the Name attribute of FieldRefs):

<ContentType ID="0x0100E3438B2389F84cc3965600BC16BF32E7" Name="Config item"
             Group="Config Store content types" Description="Represents an item in the config store." Version="0">
    <FieldRefs>
        <FieldRef ID="{33F5C8B4-A6BB-41a4-AB24-69F2152974C5}" Name="Config category" Required="TRUE" />
        <FieldRef ID="{BD413479-48AB-41f5-8040-918F32EBBCC5}" Name="Config value" Required="TRUE" />
        <FieldRef ID="{84D42C64-D0BD-4c76-8ED3-0A9E0D261111}" Name="Config item description" />
    </FieldRefs>
</ContentType>

So, the main learning I got from this is to remember that the 'Name' of the FieldRef attribute needs to match the 'Name' of the Field attribute - that simple. Why did it work before? No idea unfortunately.

However, I also picked up a few more things I didn't know about, partly from Richard (this guy needs a blog!) and partly from some other reading/experimenting..

Some handy things to know about list provisioning

To make a field mandatory on a list, the 'Required' attribute must be 'TRUE'. Not 'True' or 'true' - this is one of the cases where the provisioning framework is pernickety about that 6-choice boolean ;-)
FieldRefs need an ID and Name as a minimum (which must match the values in the 'Field' declaration), but you can override certain other things here like the DisplayName - this mirrors what is possible in the UI.
You don't have to include the list .aspx files (DispForm.aspx, EditForm.aspx and NewForm.aspx) in your Feature if you use the 'SetupPath' attribute in the 'Form' element in schema.xml (assuming you don't need to associate custom list forms).
You can use the 'ContentTypeRef' element to associate your content type with the list (specify just content type ID), rather than using the 'ContentType' element which needs to redeclare all the FieldRefs.
It's safe to remove all the default 'system' fields from the 'Fields' section of schema.xml

Going further than these tips, the best thing I found on this is Oskar Austegard's MOSS: The dreaded schema.xml which shows how you can strip a ton of stuff out of schema.xml. I've not tried it yet, but I'm sure that will be my starting point for the next list I provision declaratively. If you're interested in the nuts and bolts of list provisioning, I highly recommend you read it.

Happy XML'ing..

Slide deck from my deployment talk at Best Practices Conference

noreply@blogger.com (Chris O'Brien) — Tue, 14 Apr 2009 22:20:00 +0000

Had a great time presenting at the European SharePoint Best Practices Conference last week. I've been trying to put my finger on what made it such a good conference and I'm actually not sure, but I notice that other speakers and attendees have also been full of praise, so it's not just me. The event itself was extremely well-organized with excellent content, and Steve Smith and his team did a great job of looking after us speakers.

Highlights for me on the dev track were sessions from AC, Todd Bleeker, Eric (or "Uncle Eric" as I like to think of him, with his wise words on high-performance coding :-)) and Andrew Woody, but whenever I did stray from developer content I seemed to run into a great session like Mike Watson's on SQL Server in relation to SharePoint. Similarly I heard good things about speakers like Dan McPherson doing innovative sessions on the Information Worker track which I was disappointed to miss. [UPDATE: Here's a gratuitous shot of me in my session:]

Another highlight was being on the two dev panel sessions we did, and having an interesting debate in one of them with Todd on approaches for provisioning - declarative (Features) vs. programmatic (code/PowerShell etc.). This was probably a good lead-in to my talk the next day, and some folks came up to say they really liked this conversation and that we covered it from angles they hadn't considered, which was good to hear. [UPDATE: Photo below of the second session, chaired by AC and with (from left to right) Todd Bleeker, Stacy Draper, Maurice Prather, Andrew Woodward, Ben Robb, Brett Lonsdale, me (with the mic) and Eric Shupps:]

So all in all, a top conference, and fantastic to catch up with so many friends. Here's the link for my deck:

Slide deck - Approaches and best practices for deploying SharePoint sites through multiple environments (dev, QA, UAT, production)

Command-line support for Content Deployment Wizard now available

noreply@blogger.com (Chris O'Brien) — Thu, 26 Mar 2009 00:11:00 +0000

I'm pleased to announce I've now completed initial development on the next version of the Content Deployment Wizard - this is a beta release for the next few weeks so if you need it "just work", you should continue to use the previous version (1.1), but I'm hoping some people out there are happy to test this beta. The tool has become fairly popular as a 'handy tool to have in the SharePoint toolbox', and hopefully this release extends it's usefulness significantly for some scenarios. If you're not familiar with the tool, it provides a way to import/export site collections, webs, lists, and files or list items, either between farms or between different sites in the same farm - the Codeplex site has more details. As previously mentioned, the key new additional functionality in this release is:

Command-line support
Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use
An installer

Having command-line support for the Wizard means that it can now be used in an automated way. Some key scenarios I think this might be useful in are:

Continuous integration/automated builds - if your site relies on SharePoint content, you can now move 'real' data as part of a build process, copying selected content from 'dev' to 'build' or 'test' for example. I often see static data (perhaps from an XML file or Excel spreadsheet) used in this way in nAnt/CruiseControl/MSBuild scripts, but for frequently changing data (config values, lookup lists etc.), this doesn't work so well as there is always a static file to maintain separately.
Deployment scripts - if you have deployment scripts to 'bootstrap' a website on developer machines, again pulling real data from a central 'repository site' can help here.
As part of a production 'Content Deployment strategy' - since out-of-the-box Content Deployment is restricted to deploying a web as the smallest item, the Wizard could be used to deploy selected lists/list items/files

Obviously you might have your own ideas about where it could slot into your processes too.

How it works

First, we select the content to move as we would normally using the Wizard..
..and select the options we want to use for this export..

On the final screen, the new 'Save settings..' button should be used to save your selections to an XML file:

This will then give you an XML file which looks like this:

<ExportSettings SiteUrl="http://cob.publish.dev" ExcludeDependencies="False" ExportMethod="ExportAll"                  IncludeVersions="LastMajor" IncludeSecurity="None" FileLocation="C:\Exports"                 BaseFileName="BlogSubwebAndPageTemplates.cmp">        <ExportObjects>          <DeploymentObject Id="b0fd667b-5b5e-41ba-827e-5d78b9a150ac" Title="Blog" Url="http://cob.publish.dev/Blog" Type="Web" IncludeDescendants="All" />          <DeploymentObject Id="cfcc048e-c516-43b2-b5bf-3fb37cd561be" Title="http://cob.publish.dev/_catalogs/masterpage/COB.master" Url="_catalogs/masterpage/COB.master" Type="File" IncludeDescendants="None" />        <DeploymentObject Id="670c1fb3-12f3-418b-b854-751ba80da917" Title="http://cob.publish.dev/_catalogs/masterpage/COBLayoutSimple.aspx" Url="_catalogs/masterpage/COBLayoutSimple.aspx" Type="File" IncludeDescendants="None" />       </ExportObjects>     </ExportSettings>    

So we now have an XML 'Wizard deployment settings file' which has the IDs of the objects we selected and the export options. We'll go ahead and show how this can be used at the command-line, but note also these settings can also be loaded into the Wizard UI on future deployments to save having to make the selections again - the key is the 'Load settings..' button on the first page (which we didn't show earlier):

For command-line use of the Wizard a custom STSADM command is used. We pass the settings file in using the -settingsFile switch. To run the export operation we showed above, our command would look like:
```
stsadm -o RunWizardExport -settingsFile "C:\DeploymentSettings\ExportBlogSubwebAndTemplates.xml" -quiet
```
The -quiet parameter is optional, and suppresses some of the progress messages which are returned during the operation.
For an import operation, we follow the same process - go through the Wizard and select the settings for the import operation, then click 'Save settings..' at the end to get the file (N.B. note the 'Import settings' screen has been simplified slightly from previous versions):

The command to import looks like this:

        stsadm -o RunWizardImport -settingsFile "C:\DeploymentSettings\ImportBlogSubwebAndTemplates.xml" -quiet      

So that's both sides of it.

Using it for real

In real use of course, you may be deploying from one SharePoint farm to another. In this case, you also need to deal with copying the .cmp file from the source environment to the target if you're going across farms - if you have network access between farms (e.g. you're using it internally for automated builds/CI), a simple XCOPY in your scripts is the recommended way to do this. For production Content Deployment scenarios with no network connectivity, what I'm providing here will need to be supplemented with something else which will deal with the file transport. Clearly something web service based could be the answer.

Summary

Using the Wizard at the command-line may prove extremely useful if you need to move any SharePoint content regularly in an automated way. In contrast with other ways you might approach this, the XML definition file allows you to choose any number of webs/lists/list items/files to move in one operation, which may suit your needs better than shipping items around separately.

This is very much a beta release, but as a sidenote I'm expecting the initial issues to mainly be around the installer rather than core code - hence I'm providing a 'manual' install procedure which will get you past any such issues (see the readme). Needless to say, all the source code is also there for you on Codeplex if you're a developer happy to get your hands dirty. As I say, I'm hoping a couple of friendly testers will try it out and help me iron out the wrinkles - please submit any issues to the Codeplex site linked to below.

You can download the 2.0 beta release of the Wizard (and source code) from:

Wizard 2.0 beta release - http://spdeploymentwizard.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=25271
Wizard Codeplex site homepage - http://spdeploymentwizard.codeplex.com

Update on next version of Content Deployment Wizard

noreply@blogger.com (Chris O'Brien) — Mon, 09 Mar 2009 22:33:00 +0000

Generally I only ever talk about SharePoint tools I'm working on once they're 100% complete and ready for use, but recently I had a conversation with someone at a user group which made me think about a policy change. Regular readers will know the main tool I'm associated with is the SharePoint Content Deployment Wizard which has become fairly popular (over 7000 downloads) - occasionally I've mentioned that one goal was to implement a command-line version, since this opens up all sorts of deployment possibilities. However I've not talked about this for a while, and just recently I've spoken to a couple of people who assumed I dropped this/didn't have the time to look at it, so here I am to tell you this is not the case!

For anybody that cares, the good news is I've actually been working on this since December interspersed with blogging, and am nearly done. The yicky refactoring work is complete, and I got chance to write the custom STSADM command on the front of it on the flight to the MVP summit last week. I need to do more testing first, but I'm hoping to release a beta to Codeplex over the next couple of weeks - if you're interested in the idea of scripted deployment of specific sites/webs/lists/list items between sites or farms (remember MOSS Content Deployment only does sites/webs and requires HTTP(S) connectivity), I'm hoping some friendly beta testers will help me screw the last bits down. The key aspects of this release are:

Command-line support
Support for saving of import/export settings to a file (in the Windows Forms app) for later re-use

Shortly after this release, I'm hoping to add support for incremental deployments (so only the content which has actually changed in the sites/webs/lists/you select will be deployed), but that's not going to make into this next cut unfortunately.

Keep tuned for further updates :-)

Other stuff

Whilst I'm at it, other things in the pipeline from me include:

Blog article on optimizing WCM sites - lots of good guidance on this has been around for a while, but I'm compiling stuff from different sources and highlighting what had a big impact for us on a recent trouble-shooting/optimization piece
Best Practices presentation - as previously noted, I'm speaking at the European SharePoint Best Practices Conference in London in April. Happily, I'm now delivering my preferred talk - "Approaches and best practices for deploying SharePoint sites through multiple environments (dev, QA, UAT, production)" so that should be fun

Needless to say, there are plenty of other blog articles on my 'ideas list' too.

Sidenote - reflecting on 2 years of SharePoint blogging

Bizarrely, I'm into my 3rd year of SharePoint blogging now. I've no idea how this happened. Having done some interesting work with SharePoint's Feature framework, the initial idea was to write 4 or 5 articles I had material for - as a record for myself more than anything - and be done with it. Since then, although I do write the odd 'easy' post (like this one), generally my articles seem to take a long time to get completed, but I know they could be better. Occasionally I get reminded of this! So there's a long way to go for me to become a better blogger, but I'm fully hoping to still be at it in another 2 years time - and I'll have plenty more to say when the next version of SharePoint approaches :-)

UK user group meeting in London this Thursday, with Q & A panel

noreply@blogger.com (Chris O'Brien) — Tue, 24 Feb 2009 12:41:00 +0000

Just a quick note to remind UK-based folks within reach that there is a UK SharePoint user group meeting in London this Thursday. There are two sessions, one of which is an open Q & A for you to bring your trickiest SharePoint questions - I'll be amongst those on the panel representing the developer side of the house, but the line-up will cover all the bases. Needless to say, if you don't get chance to ask your question during the main session, there'll probably be ample opportunity in the pub afterwards. Michael Noel's session also looks extremely interesting, with a whole host of architecture/infrastructure knowledge condensed into one easily-digestible chunk.

Details from the suguk.org - to sign-up, use the link at the bottom of this post:

Session 1 - Building the Perfect SharePoint Farm: A Walkthrough of Best Practices from the Field - Michael Noel (see books written by Michael)

SharePoint 2007 has proven to be a technology that is remarkably easy to get running out of the box. On the flipside, however, some of the advanced configuration options with SharePoint are notoriously difficult to setup and configure, and a great deal of confusion exists regarding SharePoint best practice design, deployment, disaster recovery, and maintenance. This session covers best practices encompassing the most commonly asked questions regarding SharePoint infrastructure and design, and includes a broad range of critical but often overlooked items to consider when architecting a SharePoint environment. In short, all of the specifics required to build the 'perfect' SharePoint farm are presented through discussion of real-world SharePoint designs of all sizes.
• Learn from previous real world deployments and avoid common mistakes.
• Plan a checklist for architecture of SharePoint environments of any size.
• Build the 'perfect' SharePoint farm for your organization.

Session 2 - SharePoint Q & A Session

Following the session from last year we thought it would be a good idea to have a session where you can bring your SharePoint problems and hassles to and we can debate them as a group. We'll have a whiteboard, a laptop, and lots of clever people to discuss your questions and issues - so bring along your best and toughest!

The meeting is hosted at Microsoft in Victoria - arrive 6pm for a 6:30pm start:

Microsoft London (Cardinal Place)
100 Victoria Street
London SW1E 5JL
Tel: 0870 60 10 100

To register, simply reply to this thread leaving your full name - http://suguk.org/forums/thread/16904.aspx

Look forward to your questions :-)

Extending the web part framework - part 2

noreply@blogger.com (Chris O'Brien) — Tue, 10 Feb 2009 19:24:00 +0000

In part 1, I showed how we implemented a 'toolbox' of page templates and functionality modules wrapped up in a governance framework, to fulfil our client's requirement of a flexible WCM platform for building 80-100 internet sites with varying requirements. In this post, I want to detail some of the issues we ran into and the resolutions we found, focusing primarily on the 'module framework' we developed which is heavily-oriented around SharePoint web parts.

Quick recap

The client is a large multi-national enterprise, and the idea is that content authoring teams in 80-100 countries will take what we've delivered on MOSS to create their country's internet presence e.g. .com, .co.uk, .fr, .es etc., replacing the existing mish-mash of sites on different technologies with inconsistent branding/look and feel.

In terms of the module framework, the cornerstones of our implementation were (see part 1 for more complete details on these):

Module matrix - rules for which module can be used where, to guide authors away from building a user experience which doesn't 'make sense'
SmartPart-like approach, but with web part properties - web parts wrapping user controls but also supporting web part properties exposed in custom tool parts
Base web part/base tool part class - responsible for 'framework' behaviour such as checking if the current web part can be added (according to the module matrix)
Combine interface of publishing field controls with web part storage - since publishing field controls (e.g. RichHtmlField) must be added in a 'static' manner at design-time but our authors can add controls dynamically at run-time, we developed custom controls which combine the rich functionality of the publishing HTML editor with web part storage
Control adapter for WebPartZone for accessibility compliance - to get round the problem of all the HTML tables generated by SharePoint's web part framework, which will prevent a site validating for AA
Present only our web parts in the web part picker - since standard SharePoint web parts are not used anywhere in these sites
Remove unnecessary options when editing web part properties (tool parts) - to avoid confusing the authors

Issues and resolutions

I think that many of the challenges we faced are worth sharing as they came about through general web part development, rather than anything specific to what we did. Before I detail the actual gotchas, take note of some key development characteristics of our project:

Solutions and features used to deploy artifacts such as page layouts, content types etc.
Kivati Studio used for some other deployment aspects
Main functionality implemented in user controls - web parts were effectively thin wrappers around the .ascx files using LoadControl()
Web parts which are 'mandatory' are added to pages using the AllUsersWebPart element in a feature (though as the points probably illustrate, we looked at numerous ways of dealing with this)

Finding #1 - web parts outside of web part zones cannot be edited

The reason we wanted to have web parts outside of zones (perfectly possible by dragging a web part directly into page layout markup in SharePoint Designer) is for 'fixed' page modules which could not be removed by the content author. When we placed web parts outside of web part zones, we found the web parts would run fine in presentation mode but unfortunately cannot be edited (e.g. to edit web part properties) - the edit menu for the web part simply does not appear. I speculate this is because it is web part zones which are linked to web part storage, and thus web part properties cannot be persisted without a zone (the values in the markup will always be used). Hence, if you want editable web parts, you need web part zones.

Resolution - ensure all web parts (even ones which cannot be removed) live in a web part zone.

Finding #2 - embedding web parts into user control markup appears to be problematic

We tested various permutations of using web parts in/out of web part zones, and also with the HTML markup directly in the page layout .aspx or in a child .ascx file. After establishing that web part zones were required, we also found that whether the markup was in the .aspx or .ascx appeared to make a difference. This was unexpected, but the net effect seems to be that if you insert the web part markup into a web part zone which is in a user control rather than directly in the page layout .aspx (i.e. by refactoring the HTML markup for the web part zone and it's contents into a user control), again the edit menu will not display. I'm not sure why this is, but it could be related to the page execution lifecycle.

Resolution - accept that if web part zones will have web parts added to them at design-time by markup, the web part zone declaration cannot be in a user control.

Finding #3 - when using AllUsersWebPart element, duplicate web parts appear if the feature containing your page layouts is reactivated

Having decided our 'fixed' web parts would be added to pages using the AllUsersWebPart feature element (N.B. using this approach, 'default' web parts are associated with page layouts in the feature which deploys them. Web part zones are left empty on the page layout, and SharePoint provisions the web part into the zone at the time of creating a page from the layout). The issue we had with this is that all the web parts in all the zones in existing pages would be duplicated if the page layout feature was reactivated - this is because this XML is used both when the feature is activated (in the same way as say, provisioning for content types happens on activation) but also when new pages are created from a page layout.

Resolution - write a script (a Kivati task in our case) to remove duplicate web parts across all sites

[UPDATE - Waldek has an elegant solution to this problem in 'Preventing provisioning duplicate Web Part instances on Feature reactivation', as well as sample code similar to what we wrote for our script. DOH!]

Finding #4 - duplicate web parts can also appear when the page layout is customized (ghosted)

I'm not exactly clear on the reasons why customized files would ever cause duplicate web parts to appear, but that's certainly what we seemed to find. What happened is that we would deploy our master pages/page layouts using a feature to our QA environment, but immediately these files would be provisioned in that site as customized (i.e. the content in the content database), instead of being uncustomized and referenced on the filesystem. After further investigation, we traced the cause of this unexpected behaviour to the use of these attributes SPD adds to page layouts:

meta:progid=”SharePoint.WebPartPage.Document” meta:webpartpageexpansion=”full”

Resolution - ensure the version of the file does not contain these attributes. We actually switched to running uncustomized master page/page layouts even in our development farm. This means that we deployed the files using a feature and thereafter never opened them in SPD (editing only the source-controlled feature file instead).

Finding #5 - avoid setting default properties in the web part definition file (.webpart)

A final lesson we learnt is that, when working with web parts it's often better to avoid using the .webpart definition file extensively for setting default property values. There's nothing wrong with the mechanism - effectively these values are read whenever the web part is provisioned on a page, and your instance will set it's properties to these values. The problem, of course, is when you realize a property value you defined in the .webpart file needs to be updated because something changed. What happens to all the existing instances on pages around your site? As you might guess, the answer is nothing - unless you take steps to update those also, which generally means writing some kind of script to use SPLimitedWebPartManager. This can be pretty inconvenient when all you wanted to do was quickly change a default value.

Resolution - consider ensuring .webpart files are stripped to the bare minimum (assembly name etc.) and configuration comes from somewhere else. We typically rolled these config items into our use of the Config Store.

Summary

We ran into a few unexpected gotchas when building on the web part framework, but steps can be taken to minimise their impact. Hope you find these useful if you do web part development. Special thanks to Karoly Szalkary for helping to refresh my memory on some of these!

P.S. After 2 years writing about it, I've decided I no longer need to capitalize the 'f' in 'feature' - I think we're all on the same page on that one now ;-)

Extending the web part framework - part 1

noreply@blogger.com (Chris O'Brien) — Sat, 31 Jan 2009 23:31:00 +0000

Today I want to show some of the interesting things we've been doing with web parts for one of our clients. There's quite a lot to talk about so it will be over two articles:

Part 1 - background and implementation
Part 2 - issues and resolutions

There are a couple of things in particular which I think are quite cool, as we've effectively combined classic WCM (publishing) site functionality with a customized implementation of the web part framework. The context is a fairly large roll-out to an enterprise client, but what we're rolling out is a centralized platform for 80-100 internet sites. The idea is that content authoring teams in 80-100 countries will take what we've delivered on MOSS to create their own sites - replacing the existing mish-mash of sites on different technologies with inconsistent branding/look and feel.

Clearly a key challenge here is satisfying the diverse needs of so many stakeholders. So a cornerstone of the platform is that sites can be tailored somewhat, so each country has some flexibility to communicate with their audience in the way they think is best. We effectively give the authors a set of page templates and building blocks, and a system which governs how the blocks can fit together so that the user experience will still 'make sense'. Needless to say, a lot of analysis and consideration has gone into this - both in terms of what functionality was needed but also user journeys and navigation through the site, and the experience architects on our side (LBi) played a vital role here. There are many aspects to the project I could zone in on, but since I want to focus on the implementation details here, I'll briefly list some of these building block requirements before showing how we did it.

Key requirements/challenges:

In order to create the different page types, we needed around 15 page layouts, including these:

Home page
Channel hub/Alternative hub/Sub home - these are different template options for '2nd and 3rd level' pages
Content page
Product page
Media release
List - provides links to a series of related pages
Etc.

And whilst some aspects of page functionality was 'fixed' on the template, there were many other items which were optional - these were to be added to pages by the authors, either in a 'web part' kind of way or perhaps something else. Some examples of these optional 'page modules' were:

'Hero' feature - used to highlight something on prominent pages with an image/flash/text
Right-hand promo
Content editor module - allows an author to enter arbitrary content, but for reasons which will become clearer we developed an interesting custom control which is kind of a cross between a publishing HtmlField and a Content Editor web part (covered later)
Generic content module - rolls-up formatted content/links to a selected page
List/tabbed list - provides links to a series of related pages
Dynamic share price - displays latest stock price based on web service call
Product selector - using AJAX cascading dropdowns to filter products
Etc.

Although there were lots of other challenges (such as multi-lingual content, packaging/documenting every deployment aspect so the hosting company could deploy etc.!), I felt that building the 'framework' could be more challenging than individual functionality bits. To help frame what you'll read next, some initial questions we had for the implementation were:

How do these optional bits of functionality get added to the page? As web parts, or something else?
How do we get accessibility-compliance if web parts?
How do we provide configuration if not web parts?
How do we restrict which modules can be used where (as per the specification)?
Since we're in a 'flexible' publishing site, how do we determine which fields are needed on the content types? Does each content type need to have all the possible fields the author might choose to add?
If we are working with publishing controls, how would we bind the dynamically added control to the 'back-end' publishing field on the content type?

The implementation

As well as the optional page modules, most of the templates had a classic set of publishing fields. After looking at custom approaches, we concluded the web part framework had a lot going for it for the optional stuff - clearly we could avoid building a user interface to pick the module from a list/add to page/allow configuration of properties specific to the module, and also get drag and drop (amongst other things) as an added bonus. The concept of web part zones - as a container where one or more modules could be placed - was also important to our page structure.

Another challenge for the optional modules was where to store the data. If they were publishing fields, we would need every possible module to have a corresponding field on every possible content type, and this was pretty impractical when looking at the spec. Web parts, of course, use a different model and the framework takes care of data storage regardless of how many controls are on the page.

On the downside, a key thing to remember with web parts in publishing pages is that web part data is (by definition) not stored in publishing fields, and therefore isn't versioned in the same way. After discussing with the client, in our case this proved to not have as big an impact as we initially thought, due to the split and nature of what content would be stored in publishing fields vs. what would be web parts. So, having the client's acceptance of this trade-off, we went with web parts and came up with these solution elements:

Module matrix

This comprised two SharePoint lists which contained the 'rules matrix', to enforce the design team's specification of what functionality could be used on which page type. Effectively the data provides the mapping of modules and page layouts. Being list data, it meant that it could be easily updated by the central team if a policy change was required. This data was consumed by our base web part (point 4).
SmartPart-like approach, but with web part properties

We wanted the actual functionality of our web parts to be implemented in user controls, for the typical reason of avoiding building HTML in C# code (wrong on so many levels!). This is obviously what the SmartPart does using LoadControl(), but we had the additional requirement of needing to pass web part property values to our user controls - this meant we could use the familiar 'tool part' interface (i.e. setting web part properties in the right-hand pane) for control configuration.

In our model, each user control has a corresponding 'wrapper' web part/tool part which understands which properties are required and how to build the properties UI. In the web part's OnInit() method, values are passed from the web part properties to the user control so that the latter is initialized ready to do it's processing.
Base web part/base tool part

All our web parts/tool parts were derived from our custom classes which abstracted some responsibilities. Since we couldn't easily change the web part picker screen to only display appropriate web parts for the zone the author had selected, we built the check into the base web part - if an 'invalid' web part was added, the web part renders nothing in presentation mode but in edit mode we display a message to the author like this:

Adding too many web parts to a zone (count determined in the module matrix data) would have a similar effect.
Combine interface of publishing field controls with web part storage

Having decided to use web parts for our control architecture, we had one requirement for something similar to the standard Content Editor web part (CEWP). However, this control is pretty lame compared to the MOSS publishing HtmlField, and we quickly established our client needed more than the basic CEWP. So we combined the bits we wanted from both - the front end control used by the publishing field type (the RichHtmlField control), but the backing store of web part storage rather than a publishing field. This meant authors could add multiple instances of this optional module to their page (and get the nice editing experience), but because it's a web part we didn't need to worry about having a corresponding set of fields on each possible content type. In code/integration terms it's the same approach, but in the end we actually swapped the standard MOSS control for the control which fronts Telerik's RADEditor field since the client wanted to move to this:

Also note use of another control typically used with publishing fields here, the AssetUrlSelector - this provides the 'Browse...' button shown above, and can be used to provide a friendly way for an author to browse to a file.
Control adapter for WebPartZone for accessibility compliance

Since web parts normally render with a stack of nested HTML tables which won't validate against AA, action needs to be taken to remedy this if accessibility is a design goal. However this isn't necessarily a big deal - the approach is that you 'correct' the HTML for the WebPartZone control in presentation mode only, thus leaving the tables intact in edit mode for all the web part editing framework stuff which needs to happen. You do lose the client-side Web Part Services Component (WPSC) API doing this, but we had no requirement for it anyway (I rarely see it used). I initially assumed I'd have to write a control adapter to do this, but I found that David Schneider has already done the job - this works fine. It's also possible the latest version of the AKS has one, can't remember if I checked.
Present only our web parts in the web part picker

Since this is a highly bespoke WCM platform rather than a standard collaboration environment, we don't want to see any of the standard web parts in the picker for these sites. Two steps to this one:

- delete all the .webpart files from the web part galleries in the sites (N.B. we used Kivati for rolling out such changes across all the site collections - more on this in the future). However, doing this will still leave you with ListView web parts for all the lists/libraries in your site, so you also need to..
- ensure all your WebPartZone declarations have the little documented 'QuickAdd-ShowListsAndLibraries' property set to false:
```
<WebPartPages:WebPartZone id="g_AB07678E486C46bc962DFC8446A6CD13" runat="server" title="Zone 1" QuickAdd-ShowListsAndLibraries="false" />
```
Authors are then not confused by any standard web parts which aren't appropriate for our scenario:

Remove unnecessary options when editing web part properties (tool parts)

Finally, we do a bit of work with the accompanying tool parts (for properties editing) for our web parts to avoid confusing our authors with options which won't take effect. As an example, for a web part which looks like this in presentation mode:

Summary

There are many ways SharePoint's web part framework can be extended, and here I'm only showing the path we followed. For a requirement such as our client's, web parts provided a great starting point, perhaps showing there can sometimes be a place for web parts in an accessible publishing site so long as the trade-offs are understood and accepted.

In part 2 of this series we'll look at issues encountered and their resolutions.

A better Config Store for SharePoint sites

noreply@blogger.com (Chris O'Brien) — Tue, 20 Jan 2009 15:05:00 +0000

I've recently made some enhancements to my Config Store framework on Codeplex which I'm now ready to share. Many of these enhancements are a result of adapting the solution to a large project where we built a platform for 80-100 internet sites - hence it's now become a bit more 'enterprise'. With such things I generally make a deal with my employer where I do the work (or most of it) in my spare time and then get to share the code publicly, so here we go. Before I delve into the details, let's have a reminder of what the Config Store is all about:

Recap - the SharePoint Config Store in a nutshell

Regular readers may remember this is a solution which allows use of a SharePoint list to store configuration values used by your SharePoint application - the idea is that your webparts/server controls/page layouts etc. store any strings/data they need for configuration in here as a more flexible alternative to web.config or similar. Since our values are now in a SharePoint list, configuration can be updated across the farm through the browser to administrators who have the appropriate permissions. We can also optionally take advantage of all the other things lists give us such as auditing, item-level security, alerts and version history etc. Finally, a caching layer is used to avoid round trips when your code fetches configuration values.

On the last couple of projects where my team has used it, we've finished up with 100+ config items in the list - storing all sorts of things from URLs, strings and 'application behaviour' switches:

If you need a more complete overview, see:

Enhancements in the new release

Optional "hierarchical" configuration model similar to web.config

In the first release, since all the config values are stored in one list this means your configuration is stored in one 'nominated' site collection. This is fine for WCM sites which may only use one site collection, but may not easily map to certain enterprise requirements. What's new in this release is that the framework can now use a 'hierarchical' model, where a Config Store list exists in whatever site collections should have one, but a 'master' site collection is nominated which contains the 'master' config values. What happens is that if the config item you request is in the 'local' Config Store you'll get that value, and if not you'll get the value from the master list. This allows local overriding of the parent values if required - in practice we found 95% of the config items would be stored in the master list only, but having the facility to override the other 5% was critical to supporting some of the functionality we developed.

Needless to say, this is the implementation which best suited our requirements. It could be it doesn't really suit yours, but developers could consider starting with my source code and modifying, since other aspects such as the caching layer, Feature files, event handlers etc. might not need to be changed much.

If you want to continue to just use one Config Store list (even if you consume the config values in multiple site collections), the new code will continue to work just fine for this model too.
Easier to use in ASPX markup

Similar to my recent Language Store framework, the Config Store now supports easily dropping values into ASPX markup by implementing an expression builder. So if all you want to do is set the Text property of a control, you can now do with this without cluttering up the code-behind. Or, as an alternative example, here we're retrieving an URL from the Config Store (stored in Category 'PageUrls' and key 'MyAccountPage') to assign to a hyperlink:
```
<asp:HyperLink id="hyperlink1" NavigateUrl="<%$ SPConfigStore:PageUrls|MyAccountPage %>"
  Text="My account" ImageUrl="images/pict.jpg" runat="server"/>
```
Fixed caching bug for farm environments

Yes, something I have to hold my hands up to here - the caching layer in the initial release didn't adequately deal with multiple servers. The effect was that it required an app pool recycle to pick up changes to config values, rather than them taking effect immediately. Not the end of the world, but certainly inconvenient and taking away some of the benefits of using a SharePoint list for configuration. So in this release, the implementation correctly relies on a back-end resource (using a CacheDependency) to invalidate the cache across all servers in the farm. The implementation I chose was to add the items to the cache with a CacheDependency on a text file - this needs to be located on a path which all servers can access - and the event handler now updates a timestamp in the file to invalidate the item in the cache across all servers.

I went with using a file as the dependency item as I thought it was more lightweight than using a SqlCacheDependency - I didn't really want to impose the SQL configuration etc. to be able to use the Config Store. However, with the file cache dependency I've chosen, be aware that some production configurations may have internal firewalls which prevent all WFEs from accessing a shared file in this way - check before you deploy. Of course the code is there for you to modify should you wish to make changes in this area.

Amendment to Feature to prevent web.config modifications being made on Feature activation

Another thing that got in the way when I tried to implement the Config Store on our enterprise project was the Feature event receiver which adds the required appSettings keys to web.config. The idea here is that, to help simplify installation and initial setup, the required web.config entries are added with empty values so all the developer has to do is plug in the appropriate values for his/her site. Sounds great - and it is for single site collection sites. But for multiple site collections, when we come to activate the Feature in the 2nd, 3rd, nth site collection - of course the receiver runs and adds the empty entries to web.config again, despite the fact that we'd already inserted the real values on the first activation. And since the new values come later in the file, guess which ones are used?

So in this release there's a Feature property which determines if web.config modifications are made - it's set to 'False' by default but it's there if you prefer to change it.
'Config value' column is now bigger

Previously this column was a 'single line of text' but it's now a 'note'. This means you can happily store HTML/XML fragments or other larger values, which is very useful in some scenarios. However, note that if you're already using the Config Store on your site and want to 'upgrade', this schema change is significant - I discuss it in the readme.txt file.

So that's it. You can download the updates from www.codeplex.com/SPConfigStore - hope you find it as useful as we have.

Speaking at the SharePoint Best Practices conference

noreply@blogger.com (Chris O'Brien) — Tue, 06 Jan 2009 23:32:00 +0000

By now you've probably seen a few posts on the Best Practices™ SharePoint Conferences which are coming up - I'll be speaking at the European 'edition' which is being held 6th - 8th April in London, UK. I submitted a couple of sessions, but here's the one which got confirmed:

Customizing SharePoint the supported way: from end-user to admin interfaces

One of the key concepts SharePoint developers must remember is that modifications to core shipped files are typically unsupported. This session looks at ways in which you can implement the kind of customizations your users may ask for without doing anything naughty. Aspects covered include custom site definitions, modifying the site administration/central administration areas, and changing the user controls SharePoint uses with the DelegateControl architecture. The session is packed with demos, and also contains tips on the best way to make other common customizations not shown in the detailed examples.

I'll probably be off his Christmas card list for saying this, but when Steve Smith (fellow UK MVP and conference organiser) mentioned a while back that he wanted to put something like this on and I should think of sessions, I kind of assumed he meant something with the usual UK suspects, a bit like a user group meeting on steroids. It was only when the site went up I saw how wrong I was! Many of the world's biggest SharePoint names are on the speaker list - the developer track has Andrew Connell, Todd Bleeker, Eric Shupps, Maurice Prather (ex SharePoint product group) and fellow UK SharePoint MVPs Andrew Woodward and Ben Robb. And that's not all - there are 4 tracks in total and I've already noticed some non-dev sessions I'd like to get to from experts such as Spence, Mike Watson, Ben Curry and others. The keynote speaker is Joel Oleson. Regardless of your level with SharePoint, for my money there is huge potential for pushing your SharePoint capability forward at this conference.

As an aside, the other session I submitted (which I really wanted to do!) was this:

Approaches and best practices for deploying SharePoint sites through multiple environments (dev, QA, UAT, production)

Solutions and Features, STSADM export, the Content Deployment API – all can be used to move SharePoint site artifacts from one environment to another, and all have different capabilities and limitations. Failing to plan your deployment processes properly can add significant risk to a SharePoint development project, and in some scenarios the problems may only become apparent when initial development is complete and updates need to be released. In this session we’ll explore the nuts and bolts of each development/deployment approach, and you’ll emerge on a firmer footing in this complex area of SharePoint. We’ll also touch on 3^rd-party tools which can help with the deployment puzzle, from Codeplex utilities such as the Content Deployment Wizard to emerging tools such as Kivati Studio.

If you're thinking of attending and would be interested in this topic, leave me a comment and if there's a good response I'll try and persuade the organisers to shoehorn me in somewhere (perhaps in one of the lunchtime sessions which haven't yet been announced).

European Best Practices™ SharePoint Conference

Look forward to seeing you there!

Using .Net Expression Builders to set control properties

noreply@blogger.com (Chris O'Brien) — Tue, 16 Dec 2008 20:59:00 +0000

In my last post I introduced my Language Store solution for multi-lingual SharePoint sites, and showed the two ways it can be used:

In standard .Net procedural code:

string sButtonText = LanguageStore.GetValue("Search", "SearchGoButtonText");

Declaratively in HTML:

<asp:Button runat="server" id="btnSearch" Text="<%$ SPLang:Search|SearchGoButtonText %>" />

This declarative syntax is very useful as it means the developer doesn't have to clutter up code-behind files just to call the method to retrieve a value, then assign it to the 'Text' property of various controls. I've also retrofitted this to my Config Store solution (along with some other enhancements) and this will be available on Codeplex soon. You might notice it's the same syntax as the SPUrl token which can be used in master pages/page layouts to get a relative path to an image or CSS file, and that's because I'm using the same .Net technique. Since I had to do some digging to work out how this was done, I'm guessing (could be wrong here!) many other developers haven't come across this either, so here's how it's done.

Implementing an expression builder class

An expression builder is essentially a class which derives from System.Web.Compilation.ExpressionBuilder and contains logic to evaluate an expression at page parse time. The 'secret' is that the ASP.Net page parsing engine understands that it needs to call the class's method whenever it encounters an expression in the appropriate form. These are the things that join this mini-framework together:

Class derived from ExpressionBuilder which overrides the EvaluateExpression() and GetCodeExpression() methods

Declaration in web.config which associates your prefix ('SPLang' in my case) with your expression builder class

Optional use of ExpressionPrefix attribute on class for designer support

An expression in declarative HTML (as per the example above)

Taking things step-by-step, here's what my class looks like:

[ExpressionPrefix("SPLangStore")]
public class LangStoreExpressionBuilder : ExpressionBuilder
{
    private static TraceSwitch traceSwitch = new TraceSwitch("COB.SharePoint.Utilities.LanguageStore",
        "Trace switch for Language Store");

    private static LangStoreTraceHelper trace = new LangStoreTraceHelper("COB.SharePoint.Utilities.LangStoreExpressionBuilder");

    public static object GetEvalData(string expression, Type target, string entry)
    {
        trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Entered with expression '{0}'.",
            expression);

        string[] aExpressionParts = expression.Split('|');
        string sCategory = aExpressionParts[0];
        string sTitle = aExpressionParts[1];

        if ((aExpressionParts.Length != 2) || (string.IsNullOrEmpty(sCategory) || string.IsNullOrEmpty(sTitle)))
        {
            trace.WriteLineIf(traceSwitch.TraceError, TraceLevel.Error, "GetEvalData(): Unable to parse expression '{0}' into " +
            "format 'Category|Title' - throwing exception.",
            expression);

            throw new LanguageStoreConfigurationException("Token passed to Language Store expression builder was in the wrong format - " +
                "expressions should be in form Language Store Category|Item Title e.g. Search|SearchGoButtonText");
        }

        string sValue = LanguageStore.GetValue(sCategory, sTitle);

        trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info, "GetEvalData(): Retrieved '{0}' from Language Store.",
            sValue);

        trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "GetEvalData(): Returning '{0}'.",
            sValue);

        return sValue;
    }

    public override object EvaluateExpression(object target, BoundPropertyEntry entry,
        object parsedData, ExpressionBuilderContext context)
    {
        return GetEvalData(entry.Expression, target.GetType(), entry.Name);
    }

    public override CodeExpression GetCodeExpression(BoundPropertyEntry entry,
    object parsedData, ExpressionBuilderContext context)
    {
        Type type1 = entry.DeclaringType;
        PropertyDescriptor descriptor1 = TypeDescriptor.GetProperties(type1)[entry.PropertyInfo.Name];
        CodeExpression[] expressionArray1 = new CodeExpression[3];
        expressionArray1[0] = new CodePrimitiveExpression(entry.Expression.Trim());
        expressionArray1[1] = new CodeTypeOfExpression(type1);
        expressionArray1[2] = new CodePrimitiveExpression(entry.Name);
        return new CodeCastExpression(descriptor1.PropertyType, new CodeMethodInvokeExpression(new
       CodeTypeReferenceExpression(base.GetType()), "GetEvalData", expressionArray1));
    }

    public override bool SupportsEvaluate
    {
        get { return true; }
    }
}

If you're wondering why two methods are required, it's because GetCodeExpression() is used where the page has been compiled and EvaluateExpression() is used when it is purely being parsed. My code follows the MSDN pattern which supports both modes and uses a third helper method (GetEvalData()) which both call into. It's this GetEvalData() method which does the work of parsing the passed expression and then using it to obtain the value - in my case the expression is the 'category' and 'title' of the item to fetch from the Language Store. Notice that effectively, the key line in all of that is one line in GetEvalData() which calls my existing LanguageStore.GetValue() method - so effectively my expression builder is just a wrapper for this method.

My web.config entry looks like this:

<add expressionPrefix="SPLang" type="COB.SharePoint.Utilities.LangStoreExpressionBuilder, COB.SharePoint.Utilities.LanguageStore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=23afbf06fd91fa64" />

And finally here's how the component parts of the expression get used:

For SharePoint solutions, assuming we're deploying our code as a Feature/Solution we'd generally want to add the web.config entry automatically by way of the SPWebConfigModification class. You can find the code to do this in on Codeplex in the source code for my Language Store solution (in the Feature receiver).

Finally, if you're building an expression builder and this information doesn't get you all the way, the MSDN documentation for the ExpressionBuilder class has some additional details.

Enhancing the design-time experience with a custom expression editor

This is something I haven't looked at yet, but looks extremely cool! If the standard expression builder stuff wasn't convenient enough for you, you can extend things further by providing a custom 'ExpressionEditor' for use in Visual Studio. If I understand the possibility correctly, this can provide a better experience in the VS properties grid in two ways:

a custom editor sheet (e.g. a dialog to enter the 'category' and 'title' of the Language Store item - let's say 'Search' and 'SearchGoButtonText' was entered respectively, this would 'build' the string in the correct delimited 'Search|SearchGoButtonText' form required)

a custom picker using the Expressions collection - this could (I think) be used to query the Language Store list and display all the items, so that selecting the item to display the translation of is as simple as a few clicks, no typing!

I'd absolutely love to implement this for the Language Store/Config Store - so I might return to this at a later date!

Conclusion

Expression builders provide a powerful, clean way to inject method calls into your markup. In most cases we're used to seeing them return strings as in my Language Store/Config Store implementations, and note that the following implementations are also present in the .Net framework:

AppSettingsExpressionBuilder

ConnectionStringsExpressionBuilder

ResourceExpressionBuilder

However, one final thing to bear in mind is that the signature of the method returns an object - so theoretically it should be possible to do a whole host of other things, where the processsing returns a more complex object which gets assigned to the control property. An example could be data-binding scenarios where your method returns something which implements IEnumerable/IList - this could then be assigned to the DataSource property of your control declaratively. You might have other possibilities in mind, but hopefully that's food for thought ;-)

Building multi-lingual SharePoint sites - introducing the Language Store

noreply@blogger.com (Chris O'Brien) — Mon, 08 Dec 2008 23:19:00 +0000

If you're ever asked to build a multi-lingual site in SharePoint, it quickly becomes apparent that there are a few extra considerations compared to a single language site. These could include:

Information architecture
Language/culture detection
Deciding whether to use variations or not
URL strategy

..and so on. Clearly these are decisions which will have a different 'answer' for every multi-lingual project, and typically your client's specific requirements will steer your approach. However one challenge which is likely to remain constant across most such projects is this one:

How to deal with the many small strings of text which are not part of authored page content which need to be translated and displayed in the appropriate language

This is the challenge I'm focusing on here. To illustrate, here's an example from the BBC site where I've highlighted all the strings which may need to be translated but which don't belong to a particular page:

..and that's just one page - it turns out a typical site will have many of these. If you have to translate additional strings shown to authors in edit mode only, you could easily find the total number stretching into the hundreds. So we start to need a framework for storage/retrieval of these 'page furniture' items. If we were dealing with a shrink-wrapped product, .Net resource files could be a good choice, but this approach is probably not flexible enough for a website and won't allow content authors/power users to enter translations. Clearly something based around a SharePoint list is called for, so enter the 'Language Store' - my solution to the problem which you can now download from Codeplex (link at the end).

Introducing the Language Store

The Language Store is an adaptation of my earlier Config Store solution and follows some of the same principles:-

values are stored in a SharePoint list
an API is provided to retrieve values with a single method call
a caching framework is used for optimum performance
easily deployed as a .wsp

Items in the list are categorized, and have a column for each language translation:

Note that each translation column in the list is named with the convention 'LANG_<culture name>' (N.B. you might know a 'culture name' as a 'locale ID' or similar) - so when a new language needs to be added to the site, you simply create a new column with the appropriate name and add the translations. A list of culture names can be found in the MSDN documentation for the CultureInfo class.

UPDATED - the 'National Language Support' API reference page on MSDN is a better reference - think the CultureInfo page has changed since I linked to it.

Retrieving values

To retrieve a value, we simply call the GetValue() method and pass the category and title of the item to retrieve:

string sButtonText = LanguageStore.GetValue("Search", "SearchGoButtonText");

Also, since many of the items we might put in the Language Store are only used in the presentation of the page, it's often a shame to have to switch to the code-behind just to fetch these values and assign them to a control's 'Text' property. So I've provided a tokenized method similar to SPUrl, which allows you to simply drop Language Store values into your markup like this:

<asp:Button runat="server" id="btnSearch" Text="<%$ SPLangStore:Search|SearchGoButtonText %>" />

I like this because it means you don't end up cluttering your code-behind with lots of lines just for fetching values from the Language Store and assigning them to ASP.Net labels or controls. For those who don't know how this is done I'll write more about it in the next post as I think it's a cool, under-used facility in .Net.

How the Language Store determines which language to retrieve

In the current implementation, the regional settings of the SPWeb are used to determine which translations are retrieved. It's a single method in the code (a single line in fact!), so this scheme could easily be changed if you have a different requirement. We're using the Language Store on our current project, and using the SPWeb setting makes sense for us since we're building around 100 different sites in ~30 languages, as opposed to one site which displays in the local language (according to the user's thread culture or similar).

Note that if the Language Store doesn't contain a value for the requested culture, a fallback process is used similar to .Net's globalization framework:

Check preferred culture e.g. fr-CH for French (Switzerland)
Check preferred culture's parent e.g. fr for French
Check default language (determined by configuration) e.g. en for English

This is useful where some items might have a different version for say American English (EN-US) and British English (EN-GB), but other items don't require distinction so a single value can be entered into the parent column (EN).

By the way, if you're wondering where the SPWeb regional settings are because you've never needed to change them, they're here:

A checkbox allows the regional settings you make on a given web cascade down to child webs, so we simply set this at the root of the site as a one time operation.

Other bits and pieces

All items are wrapped up in a Solution/Feature so there is no need to manually create site columns/content types/the Language Store list etc. There is also an install script for you to easily install the Solution.
Caching implementation is currently based around a CacheDependency on a file - this enables the cache on all servers in your farm to be invalidated when an item is updated, but does require that all WFEs can write to this location (e.g. firewalls are not in the way).
The Language Store can also be used where no SPContext is present e.g. a list event receiver. In this scenario, it will look for values in your SharePoint web application's web.config file to establish the URL for the site containing the Language Store (N.B. these web.config keys get automatically added when the Language Store is installed to your site). This also means it can be used outside your SharePoint application, e.g. a console app.
The Language Store can be moved from it's default location of the root web for your site (to do this, create a new list (in whatever child web you want) from the 'Language Store list' template (added during the install), and modify the 'LanguageStoreWebName'/'LanguageStoreListName' keys which were added to your web.config to point to the new location. As an alternative if you already added 100 items which you don't want to recreate, you could use my other tool, the SharePoint Content Deployment Wizard at http://www.codeplex.com/SPDeploymentWizard to move the list.)
All source code and Solution/Feature files are included, so if you want to change anything, you can.
Installation instructions are in the readme.txt in the download.

You can download the Language Store and all source code from www.codeplex.com/SPLanguageStore. All feedback welcome!

Workflows not being associated with lists/libraries

noreply@blogger.com (Chris O'Brien) — Mon, 24 Nov 2008 07:59:00 +0000

I don't often do "a funny thing happened to me on the way to configuring a SharePoint farm recently"-type posts, but I'm making an exception here - this issue not only took a bit of figuring out, it's also kind of interesting! This is partly because SharePoint doesn't do what you'd expect it to do under the circumstances, and you could probably chase your tail for a while on this one if you didn't think it through.

Scenario:-

Workflow is not enabled on any of the lists/libraries in the site in production - specifically, there is no association so the workflow list is empty
The workflow we were expecting to see enabled is the standard publishing approval workflow - since we are in a WCM/publishing site scenario, this would be associated with each Pages list in the hierarchy
In our case, initial configuration had just been completed by the hosting company and we were asked to start our testing. Search configuration has not yet been performed
Previous environments did not display this behaviour
A custom site definition is used to create any site/web - this is what determines the approval workflow should be associated

At first it seemed like an issue with the site definition. I was thinking that the hosting company had missed a deployment step to STSADM upgradesolution on the .wsp containing the site definition, but accessing the servers showed the latest files in place. I knew that because search configuration hadn't been done yet, the hosting company hadn't yet created the SSP - this was their next task. I couldn't initially see a link between the SSP and workflow (since SharePoint workflows execute in the w3wp.exe process), but then I started wondering if there was any indirect link, and came up with this as a chain of dependencies:

Could it be that the sites didn't have workflow enabled because the SSP wasn't there at time the sites were provisioned? Sounds plausible, but then I thought hang on - surely what would happen is that workflow associations and so on would be there as usual, but when a workflow form is accessed it would error with the familiar message that session state must be configured:

Well, as you might have gathered, the answer is no, this isn't what happens! SharePoint genuinely will not (or cannot) add the workflow associations to your lists/libraries, and furthermore they will not magically appear when your SSP is created. You will need to go back and configure workflow on each list/library (or content type) either through the UI or with code.

So the moral of the story is:

Create your SSP before any sites are provisioned!

Simple data access pattern for SharePoint lists

noreply@blogger.com (Chris O'Brien) — Tue, 18 Nov 2008 23:38:00 +0000

So, you're in the early stages of your project and coding has started. It's already becoming apparent that some abstraction is needed for accessing data in a few key lists, but you're not sure what. Of course there's no 'one-size-fits-all' answer to this question, but let's run through some options:

Use LINQ4SP
Use the Repository pattern shown in the recently released SharePoint Patterns and Practices documentation
Build your own Data Access Layer
Do nothing - allow the calling code to do all the work

In my mind the best answer to this question is probably one of the first two - particularly on projects which are building something like a traditional application (i.e. with a domain model and entitities which are tied to data) on top of SharePoint, as opposed to projects which are doing say, simple WCM. However, like me, maybe you've still not spent significant time looking at LINQ4SP (and maybe have memories of the original LinqToSharePoint incarnation not quite making it to maturity) and are either in the same position with the P&P samples or are thinking it looks a little too enterprise for your current needs. And on a small project with a short dev cycle, I wouldn't blame you.

So onto other options - in terms of building your own DAL, well of course it's possible, but if you're on a smallish project then presumably finding the time to hand-code an abstraction for all your data needs is going to be tricky. Additionally I'm not too fond of this approach - I've just seen it done poorly by others too many times, resulting in a layer which performs badly/doesn't scale because objects aren't disposed correctly/doesn't provide the convenience it was intended to. And finally, as far as the 'do nothing' option goes, although we might not be striving for the ultimate pattern we are trying to avoid the duplication, inconsistency and maintenance nightmare that could come if we allow each developer on the project to work with the data how they like.

My suggestion:-

What I've used a couple of times is an 'in between' approach - specifically, in between doing nothing and building a complex DAL. What I'm showing here isn't exactly what I've used (made a couple of 'improvements' as I was tapping it out!), but I think of it as a very quick, lightweight pattern which at least helps reduce the worst of the problems:

public static class Employee
{
    public static readonly string ListName = "Employees";
    
    public static class Fields
    {
        public static string PersonTitle = "Person_x0020_title";
        public static string FirstName = "First_x0020_name";
        public static string LastName = "Last_x0020_name";
        public static string StartDate = "Start_x0020_date";
        public static string Division = "Division";
        public static string ID = "ID";
        public static string Salary = "Salary";
    }

    /// <summary>
    /// Fetches employee list item - notice that caller has to supply SPWeb object, but all other implementation 
    /// details (list name, fields names) are taken care of..
    /// </summary>
    public static SPListItem GetEmployeeListItem(int EmployeeID, SPWeb Web)
    {
        SPListItemCollection queryResult = executeEmployeeLookup(EmployeeID, Web);

        return queryResult[0];
    }

    /// <summary>
    /// Fetches employee DataRow for 'read-only' situations, or where we want to cache/serialize etc.
    /// </summary>
    public static DataRow GetEmployeeDataRow(int EmployeeID, SPWeb Web)
    {
        SPListItem employeeItem = GetEmployeeListItem(EmployeeID, Web);
        DataRow drEmployee = SPListItemCollectionHelper.CreateDataRowFromListItem(employeeItem);

        return drEmployee;
    }

    /// <summary>
    /// Private method which does actual query.
    /// </summary>
    private static SPListItemCollection executeEmployeeLookup(int employeeID, SPWeb web)
    {
        SPList employeeList = web.Lists[Employee.ListName];

        // query employee list..
        SPQuery employeeQuery = new SPQuery();
        employeeQuery.Query = string.Format("<Where><Eq><FieldRef Name=\"{0}\" /><Value Type=\"Text\">{1}</Value></Eq></Where>",
            Employee.Fields.ID, employeeID);

        SPListItemCollection employees = employeeList.GetItems(employeeQuery);

        return employees;
    }

    /// <summary>
    /// Example of another method related to employees.
    /// </summary>
    public static SPListItemCollection FetchAllHiresSince(DateTime StartDate, SPWeb web)
    {
        SPList employeeList = web.Lists[Employee.ListName];

        // query employee list..
        SPQuery employeeQuery = new SPQuery();
        employeeQuery.Query = string.Format("<Where><Geq><FieldRef Name=\"{0}\" /><Value Type=\"Text\">{1}</Value></Geq></Where>",
            Employee.Fields.StartDate, SPUtility.CreateISO8601DateTimeFromSystemDateTime(StartDate));

        SPListItemCollection employees = employeeList.GetItems(employeeQuery);

        return employees;
    }
}

Points of note:

We've centralized the repository details of employee data such as list name and field names

Getting hold of an Employee list item is now one line for the caller, and they don't need to know how to find the data

An SPWeb object must be passed, meaning the caller has responsibility for obtaining and disposing - more on this later

The caller has a 'get as DataRow' method - you might feel this isn't needed, but I think it can be a useful API function. In contrast to an SPListItem, a DataRow is completely disconnected from SharePoint and therefore there are no unmanaged SPRequest objects hanging off it which need to be disposed. This means it can be cached/serialized etc., and additionally can be passed around a deep stack of methods without the calling code having to be responsible for disposals (which by the way, would need to be done in a Finally block so the disposals happen even if an exception occurs)

For the 'get as DataRow' method, a helper class is used to translate from SPListItemCollection to DataTable and SPListItem to DataRow - this is principally because the existing SPListItemCollection.ToDataTable() method has a bug. (N.B. This is intentionally not in the form of extension methods for clarity!)

The class is not an instance class, so we don't do anything like wrap each list field with a property. This might not be as convenient for the caller, but means we don't have to worry about whether the data item is 'dirty'

Update operations are left to the caller

This means the calling code (in a SharePoint web context) looks something like this:

public void GetReadOnlyEmployee()
{
    DataRow drEmployee = Employee.GetEmployeeDataRow(56, SPContext.Current.Web);

    // do something/pass happily around codebase..
}

public void GetEmployeeForUpdate()
{
    SPListItem employee = Employee.GetEmployeeListItem(56, SPContext.Current.Web);
    employee[Employee.Fields.Salary] = 50000;
    employee.Update();
}

To me, this approach has a good "bang for buck" because it's extremely quick to implement but does make the situation drastically better in team development.

Passing the SPWeb object is key:-

In my article Disposing SharePoint objects - what they don't tell you, I highlight the difficulties of keeping track of objects to dispose in a complex class library. However, I'm now starting to see the code I used as an example there as something of an anti-pattern. The simplified code sample I used to demonstrate the problem was:

public void DoSomething()
{
    bool bDisposalsRequired = false;

    // get list from SPContext if we have one..
    SPList list = getListFromContext();
    if (list == null)
    {
        // otherwise get list from objects we create..
        list = getInstantiatedList();
        bDisposalsRequired = true;
    }
    
    // do something with list..
    foreach (SPListItem item in list.Items)
    {
        processItem(item);
    }

    if (bDisposalsRequired)
    {
        list.ParentWeb.Dispose();
        list.ParentWeb.Site.Dispose();
    }
}

private SPList getInstantiatedList()
{
    // can't dispose of these objects here if we're returning a list - we'll be attempting to use 
    // objects which have already been disposed of..
    SPSite site = new SPSite("http://cob.blog.dev");
    SPWeb web = site.OpenWeb("/MyWeb");
    SPList list = web.Lists["MyList"];

    return list;
}

private SPList getListFromContext()
{
    SPContext currentContext = SPContext.Current;
    SPList list = null;

    if (currentContext != null)
    {
        list = currentContext.Site.AllWebs["MyWeb"].Lists["MyList"];
    }

    return list;
}

In this structure, internal API code (i.e. the DoSomething() method) is responsible for obtaining the SPWeb object needed to find the list, which generally means it is also responsible for it's disposal. And this is where the difficulties arise. However, if the caller has to provide the IDisposable objects, it can then be responsible for calling Dispose() because it knows when they are no longer needed. Most often the caller will simply be passing a reference to SPContext.Current.Web, but if the caller happens to be a Feature receiver then SPFeatureProperties.ListItem.Parent.Web would be passed, or if no existing SPWeb object was available to the caller (e.g. console app) a new SPWeb object would have to be instantiated and passed. Disposals then become much simpler because they can always happen in the same place as any instantiations.

Conclusion:-

I'm sure there are better patterns out there, but even a simple approach like this provides much more structure than leaving it all to the caller (particularly when the calling code is being written by different developers!). For me the key thing is to standardize how code in your codebase accesses list data - much better than having field and list names dotted here, there and everywhere. One thing the approach doesn't specifically consider is unit-testing - the sample projects in the Patterns & Practices stuff look useful here, and I for one will be getting better-acquainted!

P.S. Thanks to Rob Bogue for helping me crystallize these thoughts, and apologies to Sezai Komur for not getting round to mailing a draft through earlier as I promised!

Sample code from my top WCM tips presentation

noreply@blogger.com (Chris O'Brien) — Tue, 28 Oct 2008 22:30:00 +0000

Since AC suggested I post some sample code for some of the points I was making in my top 5 WCM tips presentation, I've now done just that. In the code you can find examples of the following:

Exception handler HTTP module - sends notification e-mails when an unhandled exception occurs and redirects the user to a configurable 'friendly' error page
Custom base class for master page - this is almost a skeleton class since your implementation will vary, but demonstrates how to use a custom base class
Trace helper class and matching ReSharper templates - for logging what happens in your code (this one's a bonus since I recommend these classes implement trace!)

These files can be downloaded in a Visual Studio project from:

http://sharepointchris.googlepages.com/wcmsamplecode

But for those of you who'd just like a quick look at the code, see below. The actual classes in the project have comments to help you 'use' these things in your site - these have been removed below for readability:

Exception handler HTTP module code:

using System;
using System.Diagnostics;
using System.Text;
using System.Threading;
using System.Web;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Utilities;
using COB.SharePoint.Utilities;

namespace COB.Demos.WcmSamples
{
    public class ExceptionHandlerHttpModule : IHttpModule
    {
        #region -- Private members --

        private TraceSwitch traceSwitch = new TraceSwitch("COB.Demos.WcmSamples.ExceptionHandlerHttpModule",
                                                      "Trace switch for the COB.Demos.WcmSamples.ExceptionHandlerHttpModule.");

        private TraceHelper trace = null;

        #endregion

        #region -- Constructor

        public ExceptionHandlerHttpModule()
        {
            trace = new TraceHelper(this);
        }

        #endregion

        public void Init(HttpApplication context)
        {
            context.Error += context_Error;
        }

        private void context_Error(Object sender, EventArgs e)
        {
            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "context_Error(): Entered error-handling module.");

            // collect the last error..
            var exception = HttpContext.Current.Server.GetLastError();

            // process it/send e-mail..
            processError(exception);

            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "context_Error(): Finished handling error, " +
                "about to redirect.");

            // and finally clear the error and redirect to the friendly error page..
            HttpContext.Current.Server.ClearError();
            HttpContext.Current.Response.Clear();
            string sErrorUrl = ConfigStore.GetValue("Errors", "FriendlyErrorPageUrl");
            HttpContext.Current.Response.Redirect(sErrorUrl);
        }

        private void processError(Exception exception)
        {
            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "processError(): Entered.");

            string sMessage = buildEmailText(exception);
            trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info, "processError(): Built error string '{0}', calling SendEmail.", 
                sMessage);

            sendEmail(sMessage);

            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "processError(): Leaving.");
        }

        private static string buildEmailText(Exception exception)
        {
            var messageBuilder = new StringBuilder();
            messageBuilder.AppendLine("<strong>Date: </strong>" + DateTime.Now);
            messageBuilder.AppendLine("<br /><strong>Message: </strong>" + exception.Message);
            messageBuilder.AppendLine("<br /><strong>Source: </strong>" + exception.Source);
            messageBuilder.AppendLine("<br /><strong>Current user: </strong>" + Thread.CurrentPrincipal.Identity.Name);
            messageBuilder.AppendLine("<br /><strong>Machine name: </strong>" + Environment.MachineName);
            messageBuilder.AppendLine("<br /><strong>Url: </strong>" + HttpContext.Current.Request.RawUrl);
            messageBuilder.AppendLine("<br /><br /><strong>Exception details: </strong>" + exception);
            return messageBuilder.ToString();
        }

        private void sendEmail(String message)
        {
            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "sendEmail(): Entered.");

            try
            {
                var errorToAddress = ConfigStore.GetValue("Errors", "SendToEmail");
                var sErrorSubject = ConfigStore.GetValue("Errors", "EmailSubject");

                trace.WriteLineIf(traceSwitch.TraceInfo, TraceLevel.Info,
                    String.Format("SendEmail(): About to send error e-mail to recipient list '{0}' using SPUtility.SendEmail().", 
                        errorToAddress));

                SPUtility.SendEmail(SPContext.Current.Web, false, false, errorToAddress, sErrorSubject, message);
            }
            catch (Exception exception)
            {
                var traceMessage = String.Format("Exception thrown attempting to send error e-mail using SPUtility.SendEmail(). " +
                    "Exception Details: {0}", exception);
                trace.WriteLineIf(traceSwitch.TraceError, TraceLevel.Error, String.Format("sendEmail(): {0}", traceMessage));
            }

            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "sendEmail(): Leaving.");
        }

        public void Dispose()
        {
            // nothing to do here..
        }
    }
}

Skeleton base class for master page:

using System;
using System.Diagnostics;
using System.Web;
using System.Web.UI;
using Microsoft.SharePoint;

namespace COB.Demos.WcmSamples
{
    public class BasePage : MasterPage
    {
        #region -- Private members --

        private TraceSwitch traceSwitch = new TraceSwitch("COB.Demos.WcmSamples.BasePage",
                                                          "Trace switch for the base page class.");

        private TraceHelper trace = null;

        #endregion

        public BasePage()
        {
            trace = new TraceHelper(this);
        }

        protected override void OnLoad(EventArgs e)
        {
            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "OnLoad(): Entered.");

            // TODO: add custom code here..
            
            base.OnLoad(e);

            trace.WriteLineIf(traceSwitch.TraceVerbose, TraceLevel.Verbose, "OnLoad(): Leaving.");
        }

        /// <summary>
        /// Showing an example base page property..
        /// </summary>
        public bool UserIsAuthenticated
        {
            get { return HttpContext.Current.User.Identity.IsAuthenticated; }
        }
    }
}

My top 5 WCM tips presentation

noreply@blogger.com (Chris O'Brien) — Wed, 22 Oct 2008 21:12:00 +0000

Had a great time presenting my WCM tips presentation over the last week or so - first to a record attendance at the UK SharePoint user group (we hit 200+ attendees for the first time!) and then to a Gold partner audience at Microsoft today. I'd like to think the user group record was due in part to the great agenda, but suspect it was really down to the draw of free beer, pizza and curry from LBi ;-) Instead of posting a simple link to the slides, I want to run through the info here because:

I can attempt to convey some of the info which was in my demos here.
I know how many of you will read a blog post but not follow a link to a PowerPoint deck!

First off, this presentation isn't about discussing the standard (but critical) WCM topics which you can easily find quality information on these days. Personally I think if you're embarking on a WCM project your starting point for information should be Andrew Connell's book (and his blog articles if you don't already subscribe), and reading in detail some of the key MSDN articles which have been published on the subject (I list my favorite resources at the end). In these places, you'll find discussion on some of the sub-topics (non-exhaustive list) which I see as the bedrock of WCM:

Security
Accessibility
Optimization
Deployment

So instead of covering these in detail, my tips focus on key techniques I've found to be powerful in delivering successful WCM projects. They won't be suitable for all WCM projects, but hopefully they give some food for thought and provide some extra value to the WCM info already out there.

Tip #1 - Implement HTML markup in user controls, not page layouts in SPD

Explanation:

Instead of adding your page layout HTML to the page layout in SPD, create a 'parent' user control for each layout which then contains the HTML and child user controls for this layout. This means you have a 1-to-1 mapping between page layouts in SPD to 'parent' user controls within your VS web project. These parent user controls contain the actual markup and also any child user controls used on your page.

Slide bullets:

Faster development experience - less SPD pain [instantaneous save rather than 3-5 second save plus occasional unwanted changes to markup]
Page layout markup now in primary source control [alongside your other code]
Much simpler deployment of updates [simply XCOPY user controls when you deploy updates to other environments]

What it looks like:

So we see that our page layouts now have very little markup (note the ASP ContentPlaceholder controls must stay at the top level, so our parent user control gets referenced in PlaceholderMain):

..and then all of our real markup code is in the parent user control and child user controls which it references:

Tip #2 - create custom IIS virtual directory pointing to your web project files

Explanation:

Most project teams I see put their user controls under 12/CONTROLTEMPLATES (often in a sub-folder) so as to follow what SharePoint does. This is mandatory in some techniques (e.g. custom field controls, delegate controls), but is not required for the type of WCM page user controls discussed in tip #1, and there are arguments for not storing those in the 12 hive. In summary, having a custom IIS virtual directory pointing to your VS project means we avoid having separate design-time and run-time locations for the project files.

Slide bullets:

Store user controls/page furniture files (e.g. image/XSL) here. Remove code files (e.g. .cs files) for non-dev environments
Faster development experience – no files to copy, no post-build events. Just save and F5!
Important if using tip #1 – don’t want to have to compile project [for post-build event to copy files] just for a HTML change

What it looks like:

In our case we created a virtual directory with an alias of 'MIW' (the name of our website) which points to our project files:

All our user control/page furniture file paths then look like '/MIW/PageLayoutsControls/foo.ascx' etc.

Tip #3 - make life easier for the site authors/admins [reduce their stress and they'll be on your side]

Explanation:

This one is a non-technical tip I wanted to throw in there - whilst we're busy getting the front-end of the website right, I think it pays to also think about how authors/admins will use the back-end of the website (N.B. here I mean 'business admin' rather than IT pro). Although this is probably verging on the 'political' side of things, I'd advocate making their life as easy as possible - they often have a loud voice within the client organisation, and if they have bad feedback on what you're building that's a black mark against you.

Slide bullets:

Consider providing custom tools if the ‘SharePoint way’ is not simple enough (e.g. user management)
If you use custom lists for site data, provide a link for authors to find them (e.g. using CustomAction)
Remember these people are rarely SharePoint gurus!

What it looks like:

Clearly this will look different for every project, but for our client we created a custom Site Actions sub-menu with some key links:

This sub-menu provides navigation to a couple of key lists used to power the site, but also to some custom screens we created for user management. Here, we also did some work to simplify things by wrapping up the relatively complex process of creating a user in the membership provider, adding them to various security groups across 2 sites (we had 2 'sister' MOSS sites with single sign-on) and setting certain profile fields, into some simple screens which give a convenient summary of what just happened after creating a new user:

Finally, the 'edit profile' screens used by business administrators were adapted from those used by end users, so that the admins became very familiar with each step of the 'profile wizard' and were better able to support their users.

Tip #4 - plan for unexpected errors

This is an interesting area, partly because we're talking about that category of things which 'should never happen', but sometimes do. Having this conversation with a client (or non-technical management within your own organization) can be fun because the typical response is "whaddya mean the website going to go wrong?", but anyone familiar with software development principles knows there is no such thing as bug-free software. So the important thing is how do we handle these cases when they do occur.

There are several tips here, so I'll break them down into 4.1, 4.2 and 4.3:

Tip #4.1 - implement 'friendly' pages for 404s and unhandled errors

Explanation:

In brief, users of a public-facing website should never see a .Net error if something goes wrong on the website! If this ever does happen, the user is left with the feeling that your website is unreliable and can lose confidence in the organisation - think about it, do you ever remember seeing this kind of error on amazon.com/eBay.com/microsoft.com?

Slide bullets:

Typically use custom HTTP module to override SharePoint's default error-handling behaviour, checking for:
- HttpContext.Current.Server.GetLastError()
- HttpContext.Current.Response.StatusCode = 404

What it looks like:

On the Standard Chartered site, our 'friendly' message looks like:

Tip #4.2 - implement e-mail notifications to developers for errors

Explanation:

Sorting the user experience is one thing, but what about actually fixing the source of the problem? A key element of this is being alerted whenever a user does experience a problem, rather than relying on them reporting it. When I first told the team we'd be implementing this, I was really thinking about the UAT phase, and also that critical week or two after go-live when you'll occasionally discover some latent issue which had managed to hide itself throughout all the testing. However, what we found is that we got even more value from this in the earlier dev/testing phases. At LBi, the test team sit near the development team, so when the Outlook 'toast' popped up with an exception message which wasn't 'known' by the team, I'd use the information in the e-mail to work out which tester triggered the error, then armed with the stack trace and other information, shout over and ask exactly what they had done to arrive at the problem. Much more efficient than waiting for a full bug report at the end of the day/week!

Slide bullets:

Means an error cannot happen without the team being aware
We built this for production, but was even more useful in dev/testing!
Implemented in same custom HTTP module as error page redirection

What it looks like:

Tip #4.3 - implement proper tracing in your code

Explanation:

So being alerted about unhandled exceptions is one thing, but the stack trace and other details in the e-mail are often not enough information to actually fix the bug easily. This can be because we can't see exactly what happened in the code (e.g. values of variables) leading up to the error. The only real way to obtain this information from production servers (or anywhere where we can't use the debugger) is to add trace/log statements to your code, which when enabled will write out these details as the code executes. Since this has to be done manually, clearly the trade-off here is the time to implement this robustness. I would strongly recommend using a code productivity tool such as ReSharper (e.g. going beyond Visual Studio snippets) to drop in these statements quickly rather than relying on typing or copy/paste.

Slide bullets:

Provides ability to quickly locate bugs in your code
Trade off is time/effort to implement
Consider productivity tools such as ReSharper/CodeRush to lessen impact

What it looks like:

This is showing trace output with DebugView - notice I've configured DebugView to show me 'warning' trace statements in yellow and 'error' statements in red:

Tip #5 - design for flexibility

Again, this one is broken down into two tips:

Tip #5.1 - using SharePoint lists for configuration data

Explanation:

As I said in my presentation abstract, since the only certainties in life are death, taxes and clients changing their minds, we should always aim to develop a solution which is flexible enough to accommodate some of the changes we may be asked to implement. We're probably all used to the idea of storing site data which may change in SharePoint lists, but I like to extend this to 'configuration' information which dictates how the site behaves. The 'Config Store' framework I wrote for this can be found on Codeplex at www.codeplex.com/SPConfigStore - this provides the list, content type, caching and API to retrieve 'config items' in code. So to take an example from our project, we had a switch for whether newly-created users need to change their password on first logon. Clearly this is something we need enabled in production, but can be a pain in our test environments where the client needs to create users and then get on with running specific test scripts. So, by having such a switch in a SharePoint list, we get the benefit that key configuration of the site can be changed as easily as changing a SharePoint list item, as opposed to perhaps being stored in web.config where I'd need to RDP onto the server, open a file, trigger an app pool recycle etc.

We stored 130+ configuration settings in our Config Store list, and of course, applied appropriate permissions so that unauthorized users could not access the list.

Slide bullets:

Use SP lists for values the client may wish to edit, but consider caching
- My ‘Config Store’ solution on Codeplex does this (www.codeplex.com/SPConfigStore)

What it looks like:

Tip #5.2 - implement a custom master page class

Explanation:

Although SharePoint gets master pages from .Net 2.0, these really deal with implementing a consistent look and feel only. If you want consistent functionality or behaviour across your pages, a custom master page class can help here. To implement this, the key is to modify the class which your master page derives from in the @Master directive. We used this approach to implement code which needs to execute on every page load, and even if you don't have this requirement from the start, I'd advocate using it so you have a convenient place to put such code if the need arises.

Slide bullets:

Use for any code which should execute on every page load
Examples on our site:

Check if trial user/when trial access ends
Check if accepted Terms & Conditions
Check has supplied their initial user profile info
Enforce use of HTTPS

Can also use to expose properties for commonly checked items (e.g. username, logged in time)

What it looks like:

<%@ Master language="C#" Inherits="MyCompany.MyClient.MyProject.BasePage" %>

Conclusion/resources

So that's it for my tips, all feedback/suggestions welcome! In terms of key resources, in addition to AC's book here are some selected articles and so on I'd recommend looking into:

MSDN article: Prescriptive Guidance for WCM sites
MSDN article: How to Optimize a SharePoint Server 2007 Web Content Management Site for Performance
Accessibility Kit for SharePoint

Speaking at the UK SharePoint user group on WCM next week

noreply@blogger.com (Chris O'Brien) — Mon, 06 Oct 2008 21:20:00 +0000

Next week's UK user group is conveniently being hosted by LBi, my current employer for a Web Content Management (WCM) special. Now I could be biased here, given that the two presenters are me and my current boss, but I think we have some great sessions. In the first, we'll do a case study session partly based on the site I talked about previously in SharePoint WCM in the finance sector and Developer lessons learnt - SharePoint WCM in the finance sector, and in the second we'll talk about practical steps/best practices to consider when building such sites.

The date is Thursday 16th October, location is London (Brick Lane/Liverpool St), and we kick off at 6pm for a 6:30pm start - for folks who aren't local or can't make it down, slides etc. will be posted either here or on the SUGUK site. The link for full details and registration can be found at the end of this post - the full rundown on the sessions is:

Session 1 : Web Content Management in MOSS, real life case study – Riaz Ahmed
This session will showcase how SharePoint content management has helped the world’s most international bank, a top 25 FTSE 100 company, deliver an online digital solution for clients, analysts and employees.
See how SharePoint has helped empower analysts worldwide to create, publish and share research information. See how SharePoint has helped clients and investors get access to market information to help decide their next multi-million pound deal. See how flexible, extensible and customisable the SharePoint web content management platform can really be.

Session 2 : Best practices/lessons learnt: building MOSS Content Management sites – Chris O’Brien

This presentation explores key tips and developer techniques for building WCM sites, based on the experience of delivering sites such as that shown in the earlier session. As well as key topics such as security, optimisation and deployment, much of the presentation will cover practices which are not widely documented but which can shave time and make the difference on a very tight timescale. And, since the only certainties in life are death, taxes, and clients changing their minds, we’ll examine how to build a site which is as flexible as possible whilst not being substantially more difficult to implement. Finally, we’ll wrap up with a discussion on how developers can keep their sanity by being able to fix bugs quickly (especially when the boss is stood over their shoulder!) regardless of where they occur in the code, before going on to open Q & A.

For full location details and to register see http://suguk.org/forums/thread/13880.aspx.

As usual, we'll be going for a pint to continue the chat afterwards - all are welcome!

SharePoint dev strategies - it's not all about Features!

noreply@blogger.com (Chris O'Brien) — Sun, 28 Sep 2008 22:04:00 +0000

Something I've been meaning to discuss for a long time is the decision to develop SharePoint artifacts using Features or some other approach. I actually discussed this back in May 2007 in my post SharePoint deployment options : Features or Content Deployment?, but feel it's a topic worth revisiting/expanding on as I often see teams developing with Features without fully working out what exactly they are getting out of this approach. As you can guess by the article title, I'm not sold on the idea of using Features all the time (readers who have followed this blog from the start might find this surprising given I wrote many articles on how to work with Features) and want to put forward some points to consider when working out whether you need them or not.

Let's first consider some (selected) characteristics of Features:

Provide a means of deploying SharePoint artifacts such as list templates/site columns/content types to multiple environments (e.g dev, test, production)
Currently the only way to deploy such artifacts across multiple site collections
Require some extra overhead to create, even with the community tools available (in comparison with creating artifacts directly in the SharePoint UI)
Little/no support for certain key updates (e.g. updating a content type which has already been deployed and is in use) - updates must be done through the user interface or the API, since modifying the original Feature files to make changes is unsupported.

Given these points, one scenario I really can't see the benefit of Features for is when the solution consists of just one site collection - which is often the case for WCM sites. Why go through the extra hassle of packaging up artifacts into Features and be faced with difficulties managing updates when the artifacts will only ever exist in one site collection anyway? Sure, they may need to be deployed between environments but we have other ways of doing that.

N.B. The same applies to site definitions - why go to the trouble of creating a custom site definition when only one site will ever be created from it?

The alternative

If you aren't forced into using Features to deal with multiple site collections, not using them could be the 'most valid' choice. In my recent WCM projects, I haven't used Features for anything which doesn't require a Feature (e.g. a VS workflow, a CustomAction etc.) for a long time now, including the project I discussed recently in SharePoint WCM in the finance sector and Developer lessons learnt - SharePoint WCM in the finance sector. Certainly given the extremely tight timescale on that project, I actually feel we could have failed to deliver on time if we had used Features.

Instead, my approach is to create a blank site in the dev environment, and do all the list/site column/content type/master page development there using the SharePoint UI and SPD. My next step (perhaps not surprising to regular readers) is to use my Content Deployment Wizard tool to move all the SharePoint artifacts to the other environments when ready. Equally, you could choose to write your own code which does the same thing using the now well-documented Content Deployment API. You'll need to deal with any filesystem and .Net assets separately (generally before you import the SharePoint content on the destination), but in my view we've at least drastically simplified the SharePoint side of things. This seems to work well for many reasons:

More efficient since no development time lost to building Features
The update problem described earlier is taken care of for you (by the underlying Content Deployment API) - as an example, add a field to a content type in dev, deploy the content which uses it and the field will be added on the import site
Concept of a 'package' is maintained, so .cmp files produced by the Wizard can be handed to a hosting company for them to import using the Wizard at their end. I hear of quite a few people doing this.
We can store the .cmp files in source control and use them as part of a 'Software Development Lifecycle' approach. My approach (and I'd guess that of others using the tool in this way) is to store the .cmp file alongside the filesytem files such as .ascx files for the current 'release', and import them as part of the deployment process of moving the release to the next environment.

As an aside, when I decided to write a tool which would simplify dealing with dev/QA/UAT/production environments on SharePoint projects, I was initially torn between 'solving the content type update problem' and something based around the Content Deployment API. One reason why I decided on the latter was because the CD API already seemed to have solved the other issue!

Now I'm certainly not saying it works perfectly every time (it doesn't, though is much improved following SP1 and infrastructure update), but in my experience I seem to spend less time over the course of a project resolving deployment issues than I would do building/troubleshooting Features. Additionally, using Content Deployment allows deployment of, well, content - if your solution relies on pre-created publishing pages or you have a scenario such as your client creating some content in UAT which needs to be moved to production before go-live, Features won't help you here. The Content Deployment mechanism however, is designed for just that.

Where do Solutions (.wsp) fit in all this?

So to summarize the above, my rule of thumb for projects which aren't built around multiple site collections is don't use Features for things which don't absolutely require them. So where does that leave Solution packages (.wsp files) - should they be abandoned too? Well no, definitely not in my view. Solutions solve a slightly different problem set:-

Deploying files to SharePoint web servers such that each server in a farm is a mirror of another. Ensuring all web front-ends have the same files used by SharePoint is, of course, a key requirement for SharePoint farms - this applies to Feature files when using them, but also to assemblies, 12 hive files etc.
Web.config modifications e.g. the ‘SafeControls’ entry required for custom web parts/controls
Code Access Security config modifications e.g. those required for controls not running from the GAC
Some other tasks, such as deployment of web part definition files (.webpart)

Really, there's nothing stopping you from doing all this manually if you wanted to (especially if you're always deploying to a single server, so there are less things to keep in sync). But the point here is that Solutions genuinely do make your life easier for comparatively little effort, so the 'cost/benefit' ratio is perhaps different to Features for me - the key is using one of the automated build approaches such as WSP Builder. So, my recommendation would generally be to always use Solutions for assemblies, 12 hive files etc., particularly in multiple server farm environments.

Conclusion

My rules of thumb then, are:

Consider not using Features (and site definitions) if your site isn't based around multiple site collections - using the Wizard or some other solution based on Content Deployment can be the alternative
Use Solutions if you have multiple servers/environments, unless you're happy to have more work to do to keep them in sync
If you are using Features, plan an approach for dealing with updates such as content type updates

My message here possibly goes against some of the guidance you might see other folks recommend, but I'm just going on the experience I've had delivering projects using different approaches. As always, the key is to consider deployment approach before you actually come to do it!

P.S. Also remember, deploying using backup and restore is a bad idea ;-)

Source code for SP Content Deployment Wizard now released!

noreply@blogger.com (Chris O'Brien) — Mon, 22 Sep 2008 00:41:00 +0000

Back from holiday now, and hopefully with something useful for those of you who use my Content Deployment Wizard tool to move SharePoint content around. I've now released the source code for the tool onto Codeplex, so if you have an interest in the Content Deployment API or just want to see how the wizard works, it's now all there for you. This is kind of a big step for me as I saw the tool more as a "free product" (built over many late nights after Suzanne was asleep!) rather than an open source project - I figured I would release the code at some point, but intended to wait a little longer until the next release, since I'm actually halfway through changing the code over for the new functionality. However, my hand was slightly forced because the Codeplex administrators suspended the project due to me not having supplied source code - apparently this is a rule of the site which I had missed. I wasn't happy with the tool not being available for people who wanted it, so wanted to rectify this as soon as possible. So this explains why anybody attempting to download the tool over the last 3 or 4 days was unable to - sincerely apologize for the inconvenience people!

In any case, it had been on my mind because there seems to be an increase recently in people building their own tools and functionality around the Content Deployment API - it's certainly an area with a lot of scope for custom solutions, and folks occasionally drop me a line with a request for a help. So it seems right to stop being precious about it and release the code sooner rather than later. I also fixed an annoying bug which caused an unhandled exception if a user with insufficient SharePoint permissions used the tool.

Anyway, hope it's useful to someone!

Content Deployment Wizard - Codeplex site homepage
Content Deployment Wizard - Release 1.1 with source code

P.S. I have a couple of things temporarily ahead of Wizard development in the queue, but as mentioned previously the focus for next version will be to add support for command-line use/scripting. This means the Wizard can be used in scheduled/automated deployments, with the benefit of the granular selection of content to deploy which standard Content Deployment does not provide.

Developer lessons learnt - SharePoint WCM in the finance sector

noreply@blogger.com (Chris O'Brien) — Sun, 31 Aug 2008 21:49:00 +0000

So in my recent SharePoint WCM in the finance sector post, I talked about what we built and why I think the result is kind of interesting. What I want to do today is share some of the technical lessons learnt, and give a sense of what worked and what didn't. As I mentioned last time, UK-based folks will hopefully be able to gain more than I can provide here when the site gets presented at the SharePoint UK user group, meaning we'll answer any question you care to come up with, not just some of the developer stuff I want to discuss today.

Now to frame all this, it's important to consider the type of project this was - the terms mean slightly different things to different people, but to me the emphasis was on 'development', as opposed to 'implementation' or 'customization'. In code terms, we ended up with the following:

17 Visual Studio projects in total
4 Windows services
5 nightly batch processes
5 supplementary SQL tables (outside of the SharePoint db)

Not bad for 8 weeks work. As an aside, although the first number seems surprisingly high, in the technical washup I did with the team nobody thought this wasn't the right way to factor the code. This is partly explained by the fact this single project is actually part of a bigger program of projects being done for the client, and also partly by the complexity of the Endeca search implementation and batch processes we needed.

What worked well (in no particular order)

Using a 'development farm' amongst the developers - this means the content database is shared, and thus no effort is required for one developer to see the lists, site columns, content types, master pages/layouts etc. created by others. This is actually the only way to do team development in SharePoint for me, but worthwhile mentioning it as I know not all SharePoint shops do things this way.
Proper use of tracing - this is the idea of writing log statements throughout code to easily diagnose problems once the code has been deployed to other environments (e.g. QA/UAT/production). We used the standard .Net System.Diagnostics trace framework with levels of Verbose, Info, Warning and Error - this has been familiar to me for a long time but a couple of the devs were new to it and agreed it was invaluable. In particular, we had a lot of library code and it's often difficult to find logic bugs when you can't directly see the result of something on a screen. For me, tracing essentially gives you the power to find certain bugs in seconds or minutes which could otherwise take hours to resolve. Although adding the tracing code can slow down coding, to mitigate this we used..
ReSharper - at the start of the project I created several ReSharper templates to call our common code e.g. for tracing, and got all the team to download trial versions of ReSharper. This meant we could add trace statements in just a few keystrokes, meaning the 'I didn't have time to add trace!' excuse couldn't be used :-)
My Config Store framework based on a SharePoint list * - we stored over 130 'configuration items', from 'True'/'False' config switches such as 'enforce password change for users first logon,' to known URLs, to certain strings displayed throughout the site. We also found a couple of areas for improvement (e.g. field not big enough to store XML fragments!) which will hopefully make it into the next release.
Implementing logging/notifications for unhandled exceptions - I know the MS Enterprise Library a component for this, but we developed our own using a HTTP handler which sits in front of SharePoint's SPRequest handler. This means that whenever something happens in the code which we're not expecting, we get to find out about it immediately and can see the stack trace and other debug info in the e-mail. This was invaluable when the testers got to work, as it meant we could proactively deal with bugs before they even got reported. As soon as we noticed a mail with a new exception, we shout over to the particular test guy (identified by the user ID) "What exactly did you just do?" (which impressed them greatly!), so we nail the exact set of circumstances/data which caused the bug right there and then.
My Content Deployment Wizard tool * - I also played the 'deployment/release manager' role on this project, so I was probably the guy who benefited from this the most but I've actually used it somewhere on every single project I've done since building it. For releases when the team had updated x page layouts, x lookup lists and x Config Store items, the tool is invaluable for picking out just the changed items and deploying them to the other environments. For Config Store items in particular it was useful as some config is different between environments (similar to web.config keys) so you don't want to overwrite the entire list. For early releases when the team had made lots of of complex 'schema' updates (such as lots of intricate changes to site columns/lookups/content types), due to the time pressures I elected to take the 'everything will definitely work this way' route and drop the site collection on the target and import the whole thing (since no valuable data to preserve) so there are some complex deployment scenarios I still haven't fully tested personally, but with 3 environments on top of the dev environment to deploy to the Wizard was prettes tilitiy much a lifesaver.
Cross-site lookup field by SharePoint Solutions - this solves the problem that a lookup column can only lookup data in the current web. We use this for several key sets of data so we get to have one copy and one copy only. Damn useful.
LINQ to SQL - we use this for CRUD operations in our supplementary SQL tables, and the guys who used it agreed they saved significant time over the standard approach of writing ADO.Net code.

* hopefully it doesn't come across as shameless self-promotion to include these - the very reason I built them was to solve recurring problems I saw on SharePoint dev projects, and both utilities really did help us here.

Project challenges (in no particular order)

Team Foundation Server weirdness - for reasons we still haven't established, we found the .csproj file for the web project (i.e. the most critical VS project!) would be checked out whenever a developer compiled the solution. With multiple checkout enabled, this means that pretty much every developer had the project file checked out all the time, regardless of whether he/she was making any project changes (e.g. adding new classes). This meant we had many more merge issues than normal - not fun.
VM issues - for a while we thought ReSharper was the culprit here, but a VS hotfix brought more stability. A hunch says at least some of the issues are 64-bit related (our dev environment was matched to production in this respect), since often the problem would manifest itself via Visual Studio (a 32-bit application remember). Frequent VS crashes, "attempted to read or write protected memory" messages in the event log - oh joy.
Failure to identify shared code soon enough - often a concern on complex development projects when the team is working at high speed. We did daily standup meetings (similar to scrum) but I suspect we may have focused too much on issues rather than what was being 'successfully' developed. So we lost some time to refactoring to bring things back in line, but this is why I like to think of the approach as 'Dangerously Rapid Application Development' (for those who remember the term ;-))
Issues arising from sharing IP addresses on SSL - in several of our environments, we attempted to use the technique documented by Adrian Spear in To setup SSL on multiple Sharepoint 2007 web applications using host headers under IIS 6.0. I've used this successfully in the past but had some problems this time round - despite working fine in our QA environment, we had problems in other places. After carefully analyzing the differences, I worked out that this technique will only work if the SSL certificate being used is a wildcard certificate or is matched on the machine name rather than the site URL. This might be obvious to other people but wasn't to me!

Hope someone finds this useful!

SharePoint WCM in the finance sector

noreply@blogger.com (Chris O'Brien) — Wed, 13 Aug 2008 22:54:00 +0000

You might have been wondering where I've been for the last few weeks, so I thought I'd show rather than just tell. I rarely talk about specific 'daytime' projects I work on, but I guess I'm quite proud of this one and you might be interested in what we've done. I've been working with LBi (Europe's biggest design/technology agency) recently, leading a team of 5-7 developers on a WCM (Web Content Management) project for one of the global investment banks. You can see some aspects of what we've done with SharePoint in this post, and I'll follow up with some developer-to-developer info about techniques we used in the next one. Additionally, for UK-based folks I think Riaz Ahmed (my current boss and Mr SharePoint at LBi) is hoping to do a real show and tell at the UK SharePoint user group in a couple of months, where there'll be the opportunity for Q&A and to pick up far more info/lessons learnt (both technical and non-technical) than I can provide here.

The site is targeted at a closed audience over the internet (our client's clients), so unfortunately you can't see it for yourself - the site provides analysts with high-value information on financial markets, primarily in the form of market reports and newsflashes, though there are other content types. The user experience we were tasked with implementing (created by LBi's design team) means the user gets to have some pretty impressive functionality around all this (IMHO) which differentiates the service from others.

The centrepiece is the market report data, which is held for many countries and topics:

This is imported nightly from a master datasource (thus lending itself nicely to caching), and our client can also add comments to the imported data via SharePoint WCM page editing. Perhaps the slickest piece of functionality is the facility for the user to create personalized 'smart reports' with specific markets and topics which interest him/her. These are saved against the user's profile for quick access:

At runtime we build the personalized page from the same cache layer used by the standard pages in the first shot.

Since financial markets change quickly, our client's authoring teams (distributed around the world), can quickly create newsflashes via SharePoint publishing functionality for display on the site - this is controlled via SharePoint workflow. Users can search and filter by country, date, high priority flag and can also store in a 'My newsflashes' bucket for quick retrieval:

Users also have the option of receiving e-mail alerts when newsflashes are added to the site (immediate for high priority newsflashes, nightly batch for others), and can tailor their e-mail preferences by country...

..and topic:

The site also has a ton of other good stuff, such as:

single sign-on with a sister site
integration with a higher-level portal which can authenticate users itself (meaning users do not log-in separately on our site)
WCAG 'A' -level accessibility - accessibility wasn't particularly a focus of the client for this site, but (happily) is just how things are done here anyhow. We use a lot of JavaScript to enhance the user experience but it always provides equivalent functionality when disabled - I have to say the LBi front-end developers use some incredibly innovative techniques to achieve this, I've rarely seen anything like it (see below for some examples).
integration with Endeca for search (including the newsflash landing page you see above - this is powered by search)
custom admin functionality for user creation/management
many many smaller but 'interesting' requirements such as newsflashes having user-selected 'related items', attachments etc etc!

Rich, accessible controls

In addition to all the personalization, one of the things that I love about the user experience is the funky controls we have. So why have a dull dropdown like this:

...when you can have one like this:

This allows us to present only countries the user has told us they are interested in in the dropdown (thus avoiding clutter), whilst allowing them to easily see data for other countries if they have the occasional need to. To do this, clicking the 'More...' link expands the dropdown to show a second section containing the other countries (or more specifically, the other countries they are permitted to see):

Similarly for multiple selections, why have this:

...when you can have this:

Again, clicking the 'More...' link allows the user to select a country not in their 'preferred' countries:

Personally I think this makes for a great user experience, and it's all accessibility-compliant.

Project plan : caffeine

The development time for everything you see here (and everything you don't) was around 7-8 weeks, though this excludes some of the front-end development (HTML, CSS, JS). We're not quite over the finishing line yet, but are in the final stages of user acceptance testing and security testing. To say it's been tough would be something of an understatement, but I'm incredibly proud of my guys and it's been a great team effort - fortunately for me they're very talented :-)

Next time I'll detail some of the technical decisions we took and go through some developer bits which we felt worked well when building/deploying the site.