System.What?

Free Last Modified By/Date From SharePoint

2009-06-09T00:33:00.001-07:00

Who Responsible For This Edit!?

With the exception of politicians, most people are in favor of accountability. When it comes to web content management, I'd be the first to agree.

In this post we're going to go over how to get free Last Modified By and Last Modified Date fields on content pages by modifying either select page layouts or master pages. The end product will look like:

What We're Using

We'll be keying off two base fields (Modified, and Modified By) that you can expect to find on pretty much any list you'll ever work with. To prove to you I'm not making things up, feel free to look up the fields yourself (and a tonne of other useful fields) by using SharePoint Explorer for WSS v3.

Easy Win

Once you find out the names of the fields and their type, the markup to get at these fields is pretty straight forward. We use out of the box SharePoint SPField controls to pull and format the field values. It's of note that we put the ControlMode property to "Display" since we never want these fields to be editable.

Last modified at:
<%@ Register Tagprefix="SharePointWebControls" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
...
<SharePointWebControls:DateTimeField FieldName="Modified" runat="server" ControlMode="Display"/>by
<SharePointWebControls:UserField FieldName="Modified By" runat="server" ControlMode="Display" />

That's pretty much it, throw the above markup into any page layout or master page and you're pretty much done.

But Wait There's More

I'd hate it if you left this post thinking all you'd learned about were two lousy fields and how to display them on a page. There's a tonne of fields that are worth exploring and could potentially add value to your next SharePoint related application.

Example 1: ETL developers could make use of the Created/Created By/Modified/Modified By fields to perform incremental loads when pulling data from the SharePoint object model.

Example 2: UI Developers could use the Comments and Version fields to display the current version and any comments associated with the current version.

The point is that there is yet another lever here to help you get the most functionality out of your next SharePoint related application. Best of all, it's dead simple and doesn't involve a single line of code.

Best,
Tyler

CSS Selector Collisions

2009-04-10T19:52:00.001-07:00

Thick Competition

For those brave souls who actively try to make stock SharePoint UIs look user friendly, there's a gauntlet that first needs to be run. The challenge has to do with competing CSS rules and styles. Stock SharePoint will emit many style sheets who will often also be trying to style the same element that you are.

When it comes to deciding which rule wins, you should keep the following rules in mind. They'll very likely help you troubleshoot, especially when combined with tools like the IE Developer Toolbar or the Firefox Web Developer Tools extension.

Specificity

In general more specific rules tend to win. Remember that CSS styles can come from four different places. Rules that are increasingly specific will trump those who come from more generalized sources. Below we list (from most general to most specific) the four most common sources of CSS for a given html document.

The browser's default styles. Believe it or not each browser has a stock set of styles that get applied to elements whether you've chosen to style them or not. You'll have to compete with these even if they're the easiest to best.
An external style sheet (referenced from within the <head> tag [below]).
<head>
<link rel="stylesheet" type="text/css" href="mystyle.css" />
</head>
Internal style sheet (rules described in <style> tags within the HTML document itself).
<HEAD>
<STYLE type="text/css">
H1 {border-width: 1;}
</STYLE>
</HEAD>
Inline styles (styles placed on individual elements themselves).
<P style="font-size: 12pt; color: fuchsia">Some Text</P>

More specific rules (those further down the list) will win out over more general ones.

Weight

Certain selectors are a little stronger than others. More specifically, ID has more weight than Class which has more weight than Tag name.

In fact weight of a selector is determined by taking the sum of (100 * [# IDs used] + 10 * [# Classes used] + [# Tags Used].

Here's an example:

#navigation div.container span a.footerLink

Has a weight of (123 since it has 1 ID, 2 classes, and 3 tags. It would be trumped by:

div#navigation div.container span.wrapper a.footerLink

Since the above has a weight of 134 (1 ID, 3 classes and 4 tags).

Order

If specificity and weight are the same then it comes down to who was emitted last. In general rules that were emitted later on in the document will win.

The Ace In The Hole

If you're too lazy to figure out the rest you can simply use the !important flag.

div
{
color:red !important;
}

This should trump all other rules but is considered to be pretty poor form and may get you dirty looks around the water cooler.

Hope that helps.

Best,
Tyler

Is It Really My Job To Fix That!?

2009-03-05T18:43:00.001-08:00

That's Not On My Resume

I have to tell you that working in a software services firm isn't always a picnic. Being a vendor (read outsider), and responsible for delivering working solutions on other peoples infrastructure, in a foreign LAN is nothing but a brazen undertaking. It's akin to trying to construct a building in a war torn, 3rd world country that happens to be in the middle of ongoing natural disasters. The drama sometimes matches that of most reality TV series, and while it's rarely dull in some regards, there's easier ways in life to get a win.

When the hardware isn't changing and internal IT isn't ignoring you, it's most likely that you'll get bad OS builds, under powered VMs or fail to take note of the hidden agendas in a politically charged atmosphere. This is all in addition to regular project management woes. I'm just talking about delivery here, requirements and software lifecycle are another boat in entier.

You need to be really sharp to see all the possible ways you could fail, and even then it'll probably be something small that never even crossed your worried mind that will end threatening you and your project. Game on.

An Instance of Success

Today we came up with this interesting problem when installing some ETL on a BI server. The machine was running SSIS RTM on a Win2k3 x86 Service Pack 1. The ETL was taking a bunch of records from some source SQL Server database (SP2) and migrating them to another destination SQL Server database (SP2). The ETL would run for about 20ish minutes before consistently throwing the following exceptions:

An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Protocol error in TDS stream".

An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Communication link failure".

An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "TCP Provider: An existing connection was forcibly closed by the remote host. ".

Hmm...what to do. The ETL worked great in stage and in our own QA environment. After an exhaustive troubleshoot (on our client's machine) the fix ended up being...disabling TCP Chimney offloading. Not even on our BI server but on the destination SQL Server!

What!? You say? Your problems with delivery weren't even vaguely related to the technologies you were developing!? Exactly, something not even vaguely related to ETL became a huge friction point between us and a client. The worse part is that there's always a risk of this. Vendors come in and constantly take it on the chin (or sometime just look incompetent) because a dependency they depend on isn't working.

What makes it worse is you may have never even seen that machine in your life, and it's the one that just might bury you. Half the time you don't even have the necessary rights on the given machines to fix it let alone conduct a decent troubleshoot. All of this just might want to make a man take up farming.

This time we did have the necessary rights on the machine. We got lucky and there's even a chance that we might actually get paid for the troubleshoot...this time. Who knows what kind of circus fixes we'll be trying to pull off tomorrow. The good news is that after a while some partners actually start to trust you.

Hope that fix helps some vendor somewhere, we sure could have used it.

My Best,
Tyler

STSADM Properties

2009-02-18T17:11:00.001-08:00

Wait...There's Another Flag

If you've ever felt like the STSADM utility is taxed with doing more than Emacs, there's a good chance you're in the majority. I was recently trying to figure out what SharePoint settings affect the "New" icon when I stumbled upon the stsadm's getproperty and setproperty flags.

Essentially there's a bunch of SharePoint settings (some of which there's no admin UI screen for) that can be manipulated using this utility. The new icon is one such property.

It might be worth looking at some of these, they include both server and web application level properties.

Server Properties

Web Application Properties

As it turns out the default value for the new icon is 2 days. You can show and change this property by using the following commands:

stsadm -o setproperty -pn days-to-show-new-icon -pv 2 -url [WebAppUrl]

stsadm -o getproperty -pn days-to-show-new-icon -url [WebAppUrl]

This probably wont interest you in the least until you start to think about scripting and how this might help you ensure certain settings have been set when it comes time to deployment.

While many of these settings might be alterable using the SharePoint API, there are still many types of changes that deserve to be versioned in some kind of script rather than C#.

Best,
Tyler

Start Your April Fools Jokes Early

2009-02-15T23:08:00.001-08:00

Time Waits For No One

For some reason Valentines Day always reminds me that April Fools is coming up. This most likely draws from the fact that I'm the youngest of two siblings, and it's become a personality trait to express my affection by irritating the crap out of an individual.

Needless to say I have very few close friends.

Since nothing says "I care about you" like a good practical joke, I'd like to take the time to remind you that April Fools day is a very short 32 business days away, and depending on how deep your pranks run, you may need to get some infrastructure in place.

The Right Joke For The Right Bloke

A good practical joke is tailored to the individual. Knowledge of their technical depth, their patience, any history of violence, and past convictions are key to finding just the right level of crazy.

If you'd like to go shopping for ideas, I'd suggest starting here. My only advise is to be prepared for retaliation. Most pranks usually deserve another, and the minute your cover gets blow, it'll already be too late to start wearing a helmet.

Get Going

So get on it. Usually most April Fools jokes fall apart because people leave them to the last minute. DNS records may need to get updated, proxies may need to get provisioned, machines may need to get hacked. Don't risk leaving it to the last minute, start being devious and subversive today!

I'll share this years war story on April 2nd.

Best,
Tyler

How SharePoint Found Its Content Database

2009-02-09T00:02:00.001-08:00

Good Question

I was talking a client through a SharePoint web.config change a couple of weeks ago and he ended up asking a dynamite question.

How does the web application know where its content database is at?

You might say something like "it gets it from the config database", but that would beg another question, "how does it know where the config database is?". In reality, we're curious about how a web application orients itself to its environment when it comes time to respond to a request.

You'll notice that SharePoint web.configs don't have any connectionString elements in them that point to a particular content or configuration database, and SharePoint installs don't make any changes to parent web.configs or machine.configs. So how does this web application orient itself at runtime?

Lets Reflect, Literally

If you think about it, there's a few clues to where this code "ought to be". Pretend for a minute that you were tasked with authoring some piece of functionality that let a web application figure out where its database was at runtime. There's some typical places you'd hook in (besides the web.config). You'd know that this piece of functionality would need to be populated in time to serve requests who would need to know where the content db is at.

You might think about putting it in global.asax Application_OnStart or some http module event that's relatively early in the ASP.NET pipeline.

Lets look at how a typical SharePoint request gets resolved and see if we can't flush out some details. This is the path I took to piece this answer together.

A request comes in and gets handled by IIS. The request matches either a port number or a host header on a web site in the web server.
The matching SharePoint web site has wild card ISAPI mappings that pass all request to ASP.NET. Remember that in a typical ASP.NET web site there mappings for known ASP.NET file extensions (.aspx, .asmx, .ascx, etc...) to the ASP.NET ISAPI filter. When IIS gets a request that ends with the given file extension, it knows that this is an ASP.NET request and hands it off to the aspnet_isapi.dll ISAPI. In a SharePoint IIS web site, we pass everything to ASP.NET. This includes .gifs, .jpegs, .PSDs etc... SharePoint gets a chance to fulfill every requests that gets picked up by the site. Below is a screen cap the wild card mapping present in all SharePoint sites. This is created for you automatically when you provision a new web application. If you were to remove it, your SharePoint instance would no longer be able to serve non ASP.NET files out of the content database.
Once the request gets handed to ASP.NET, its fair game for the web application to start doing some magic. In fact, it had better figure it out soon cause odds are this request is going to want something from the content database before the page life cycle hits. Because we're talking about the ASP.NET request pipeline (and early in it), it's a reasonable guess to assume an http module could be in play. If we look in the web.config under the <httpModules> section we find that SharePoint has registered an HttpModule named "SPRequest", it's fulfilled by code found in the Microsoft.SharePoint.ApplicationRuntime namespace by a class called SPRequestModule.
If you reflect on SPRequestModule and look into BeginRequestHandler (one of the earlier events in the ASP.NET pipeline), it's obfuscated, but you can read it anyway. BeginRequestHandler makes a call to SPFarm.Local which ends up calling SPFarm.FindLocal in an effort to find the farm that this machine belongs to.
SPFindLocal finally calls into SPConfigurationDatabase.Local which runs to the registry looking for HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\Secure\ConfigDB. This is where your "connection string" for the current farm lives.
This is also how SharePoint finds the config database, from there it simply takes the current request and uses it to find the content database (if any) associated with the given request. After all this has been figured out, a lot of it gets cached.
Finally BeginRequestHandler starts through the complex task of mapping your request to a bunch of content in the web applications content database. The regular Page Life Cycle kicks off and at the end of the day we end up with an HttpResponse built up by SharePoint.

Thought that someone else might enjoy the answer. This is kind of a neat piece of functionality too, it allows the psconfig tool to move this machine from farm to farm without changing any web.config files, so long as you can tell it where it should remap the registry key to. And with that, there's a little less magic in the world.

Best,
Tyler

When the Reflector Tells You To Hit The Road

2009-02-06T01:18:00.001-08:00

Seeing Obfuscated Code

Like every other .NET developer out here, I'm a big fan of the .NET Reflector. It was originally written by Lutz Roeder but is now maintained by the kind gentleman at Red Gate. One of the elements that make the tool stand out (in addition to the ease at which it lets you navigate/disassembling code), is the ton of add ins on CodePlex that extend the base functionality. If you haven't already, you should explore them some time.

I mostly use the .NET Reflector when it comes to things that I don't quite understand and I can't seem to find an adequate answer on the web. Ninety percent of the time, this tool will usually make it easy for you to look underneath the hood and you can find your own answers.

Every now and then when you're reflecting on code that someone doesn't thing you should see, you'll get the following message from the tool.

This item is obfuscated and can not be translated.

If your stubborn like most rural mules or have never heard of "no means no", you can carry onward by going to View->Options->Click Dissassembler and set the language to IL.

If you're simply asking for the IL, Reflector will gladly show you the contents of the function (if you can make any sense out of it). Intermediate Language is even harder to read than disassembled MISL that's mapped to C# or VB.NET (which unreadable for another variety of reasons).

In fact if you're just going to be looking at IL, you can actually skip the .NET Reflector in the first place and use the MSIL Disassembler (ildasm.exe) that comes with the .NET framework. May not look as pretty, but you're guaranteed to have it on any machine which as the .NET SDK.

Here's a disclaimer.

It's also worth mentioning that by reverse engineering code you could be violating an EULA, and I'm hear to tell you that you should never do that. In fact, if you're ever given a piece of code that doesn't function correctly, you should never try to fix it or do anything to further your understand what's going wrong. Often the most legally sound action, is to turn off the computer and go home.

Hope that helps someone find an answer.

Best,
Tyler

Quick Deploys And Assembly Resolution

2009-01-30T00:14:00.001-08:00

The Error

We recently ran into the most bizarre error on a clients machine that hosts the central administration web site. Every 15 minutes on the dot we'd get the following error for many different assemblies. The error looked a lot like:

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Runtime
Event ID: 6611
Date: 1/15/2009
Time: 1:11:26 PM
User: N/A
Computer: [MACHINE NAME]
Description:
Error: Failure in loading assembly: [AssemblyName], Version=1.0.0.0, Culture=neutral, PublicKeyToken=e376b6bc65267f90

It's worth mentioning that these errors mentioned the names of assemblies that were used in multiple sites (even though they were deployed into bin folders). The sites that used these assemblies were customized, and their Master Pages and Page Layouts referenced the assemblies that supposedly couldn't be loaded (as far as the error was concerned. Some of these dependencies were strong named and others were not. None of them lived in the GAC (all bin folder deployed).

The Solution

We used the fact that the errors occurred every 15 minutes to link them to Quick Deploy jobs. When we changed the quick deployments to every 10 minutes, the errors followed suit reoccurring at the same interval.

At this point we were convinced that the SharePoint Timer (OWSTimer.exe) was trying to load these assemblies but couldn't find them. So how does assembly resolution happen in the .NET framework? We were saved by an an excerpt from Essential .NET, Volume 1: The Common Language Runtime, a book by Don Box and Chris Sells. Essentially it works like this.

When the assembly loader goes to load an assembly, it first looks to whether the assembly is strong named. If it IS, then the loader first looks in the GAC (not the bin folder).
If it can't find the assembly in the GAC, the loader will then look for <codeBase> hints in the applications configuration file. Remember that these settings inherit down from machine.config to app.config/web.config.
If there's no <codeBase> hints then the loader will resort to probing as a last ditch effort to find the assembly. This includes bin folders and any other location dictated by <probing> elements.

If the above seems confusing, consider the following flow chart.

We first tried putting hints in the local web.configs, but because the OWSTimer has nothing to do with the our applications, the settings were being ignored. We finally helped the timer out by making the following modifications to the machine.config. To ensure that we didn't remap ALL lookups for the given assembly to the specific location, we had any other application who used the same assembly name override these settings below with a similar one in their own web.config. The overridden setting listed a location that made more sense for the particular web application (ie. web applications should go hunting for .dll's in their own bin folder, not some other applications).

The machine.config was changed to read:

<runtime>  
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">   
   <dependentAssembly>   
     <assemblyIdentity name="[NameOfAssembly]" publicKeyToken="9871993fa258bc6" culture="neutral" />   
     <codeBase version="1.0.0.0" href="file://C:/folder/directory/bin/[NameOfAssembly].dll" />   
   </dependentAssembly>   
 </assemblyBinding>   
</runtime>

The web.config was changed to read:

<runtime>  
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">   
   <dependentAssembly>   
     <assemblyIdentity name="[NameOfAssembly]" publicKeyToken="9871993fa258bc6" culture="neutral" />   
     <codeBase version="1.0.0.0" href="file://C:/webapplicationLocation/bin/[NameOfAssembly].dll" />   
   </dependentAssembly>   
 </assemblyBinding>   
</runtime>

You can also provide <codeBase> hints for assemblies that aren't strongly named, you just need to omit the the optional publicKeyToken and culture attributes. Another tip is that you can also use the <probing> element if you want to probe folders other than the bin folder.

We then restarted IIS, followed by the timer service. Low and behold, our random error went away. This toast goes out to the highly configurable .NET framework, and the expressiveness of app/web/machine.config files.

Best,
Tyler

TroubleShooting The JQuery Accordion

2009-01-27T21:03:00.000-08:00

JQuery and the Accordion

I've been getting more and more into JQuery recently. It's light, free (as in freedom), and has a vibrant community. Sweetening the deal is the ever growing list of animated controls that effortlessly plug into the framework.

We've been using JQuery for quite a few of our clients who have expressed a desire for more interactive SharePoint sites. I've been so impressed with the framework of late that I've started to advocate it over heavier tools that try to achieve similar results.

The Problem

Just today I ran into a pretty interesting problem with the JQuery Accordion. I'm throwing up this fix because I couldn't find a solution on the web. We had the accordion binding off of a SiteMapDataSource in a SharePoint master page, everything looked great in Chrome, FireFox, and Safari...but IE 6 and 7 were giving us a really hard time.

What made this even more frustrating is that it worked fine in at least three other SharePoint sites with exceedingly similar master pages.

The accordion would flicker at the end of the animation every time the user would expand or collapse a sub menu. We tried removing all the CSS, but the page still yielded the same behavior. The menu would briefly flash the entire contents of a sub menu whenever a a new section was collapsed/expanded. The only way we could stop this was by turning the animation off in entire.

The Troubleshoot

At first I tried debugging the JavaScript for JQuery and the accordion, but the source quickly became difficult to follow. The guys who wrote this stuff have mad JavaScript skills, they're head over heels better JavaScript developers than I am.

So I ended up doing what I normally do when I don't understand what's going on. I started to make things simpler. I opened up SharePoint Designer and started removing sizeable sections of the page, block by block, seeing if I could get the problem to go away. I ended up with a ridiculously small html document that basically had the the accordion menu markup and a couple of other html tags.

It was only when the document was about forty lines long that the problem became apparent.

There was no doc type on the html document. I've written about Quirks Mode before, so I was a little embarrassed that I hadn't noticed it earlier. If you're not familiar, or are too lazy to click on the link above, in a nutshell quirks mode is a special rending mode that browsers go to when you either don't specify the doc type or use html that deviates from the declared doc type. The rules for what justifies quirks mode rendering changes from browser to browser. For IE, not declaring a doc type will cause it to render the page in quirks mode.

Quirks mode changes the way that the browser handles CSS rules. When we failed to declare a doc type, IE started rendering the menu in quirks mode. This changed the way that the browser handled the in line CSS that the JQuery accordion was tacking onto menu list markup. Hence the odd behavior.

I added my favorite and most relaxed doc type:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

Low and behold, the animation issue went away. Quirks mode strikes again. I guess the lesson here is that before you get to involved in a markup troubleshoot, ensure the document isn't being rendered in quirks mode! You can figure out how to detect quirks mode by reading this.

Fool Me Twice,
Tyler

SharePoint: Searching With Managed Properties

2009-01-22T01:01:00.001-08:00

Crawled and Managed Properties

When the SharePoint 2007 crawler indexes a piece of content, it finds a not only a bunch of text to index, but also a tonne of metadata about the document. These pieces of metadata are referred to as crawled properties. Consider an Office 2007 Word document (.docx). When the crawler opens up the document, it finds a plethora of information about the file; the file extension, who authored it, when it was created, and the actual text in the document itself. These are just some the many crawled properties captured about any given piece of content. What fields actually get captured are a function of the content and the IFilter that helped the crawler dissect the document.

Here's a reduced screen cap of some of the crawled properties captured for Office Documents. You can sift through all the properties that are being captured by your crawler by going to the Shared Service Provider, then clicking on Search Settings, Metadata property mappings and then on Crawled Properties. If you click on the Office category you'll end up getting a list that's much longer than the abridged one on the right. Whenever the crawler discovers a new property, it adds the new crawled property to a list maintained in the Shared Service provider. If the property discovered is of type text, then it is automatically added to the search index.

It's important to note that a Crawled Property can be reused in many Managed Properties, and Managed Properties can have mappings from many Crawled Properties (many to many). In fact this is how we harvest value from the Crawled/Managed property marriage. By using them together we can construct extremely powerful and expressive search queries.

Managed Properties are user created labels, that map one or more crawled properties to a searchable term. They are available for basic search, advanced search and for defining scopes. An example of a search expression using a managed property might be the search:

Title:"Tyler"

When I search Title:"Tyler" (read [ManagedProperty]:"[SearchTerm]"), I'm really searching all the crawled properties that are mapped to the managed property Title, for the text "Tyler".

That is, the search expression Title:"Tyler" will bring back all the search results that have the text "Tyler" in at least one of the following crawled properties [Mail:5(Text), People:PreferredName(Text), Basic:displaytitle(Text), ows_Title(Text)]. In fact if you feel that the crawled properties attached to Title aren't sufficient, you can make any changes you want. Managed properties are there for you to explore and tweak.

Creating a Managed Property

You don't really have a lot of control over crawled properties (you simply get what the indexer picks up). You do however get a fair bit of hand over what managed properties exist, and what crawled properties are mapped to them. It's worth mentioning that you don't always need to install a new IFilter to get more crawled properties to show up. The indexer will also pick up custom site columns that you add to your lists/libraries every time it does a full crawl. If this site column has never been seen before (ie. you just created it), a new entry will be added to the crawled properties list.

A good example of this would be if you were to create a new site column (say MyCustomeField) on some list. The next time the indexer does a full crawl, the new crawled property will show up (named ows_MyCustomeField). You can then open up Metadata Property Mappings (from the Shared Service Provider Search Settings) and create a new managed property (say TylersCustomField) that users can use to search exactly that site column. They could, for exaple search for TylersCustomeField:"SomeValue". You can even use that managed property to limit or grow the search results that are governed by any search scope.

When you start to digest the utility of Managed/Crawled properties, it becomes apparent they they're responsible for a huge part of the SharePoint search's expressiveness. They empower you to help users author some pretty functional searches. Not just the kind of searches that find all the documents with a certain file extension (fileextension:"doc", but searches that look only at very custom properties within niche types of content.

Consider the following out of the box managed properties that allow searching for people by their assistant's name (Assistant:"Mary"), by their Responsibilities (Responsibilities:"Sales"), or finding files by filename (filename:"my document"). Managed properties can also be grouped in searches (fileextension:"doc" filename:"my document").

Stop by the Managed Properties pages in the Shared Services Provider and you'll be impressed. Once you start to wrap your head around managed properties, you'll realize how easy it is to leverage this out of the box functionality for your customers. You'll may even start to feel like SharePoint is finally making your job easier.

Best,
Tyler

Turn On The Search Already!

2009-01-18T22:15:00.000-08:00

ROI Unleashed

When IT departments roll out SharePoint, search is usually the last thing on their minds. This is really a shame, because powerful enterprise search is one of the most underutilized features that comes out of the box with MOSS Standard or Enterprise. You get search with WSS too, but you can only search a given site collection.

MOSS allows you to search across a whole enterprise unifying rogue document shares, reclusive business data, and your actual SharePoint content. People can also search for People and find internal resources that possess a particular skill set, or report to a given manager.

Business units naturally tend to carve out their own space for storing tribal knowledge. This is usually irritating to system administrators and confusing to new hires (never mind existing employees). One of the easiest corrective actions is to tell SharePoint to go index all these pieces of hidden content and yield them when the appropriate search string is entered.

Creating a search portal is one of the most obvious value adds that you can quickly provide your users. I can't think of any organization I've ever worked with that didn't have documents scattered across the enterprise. In a day and age when IT is constantly trying to justify it's ROI, turning the key on enterprise search is an easy home run.

What's even better is that SharePoint Search will automatically security trim based on ACL (Access Control List) entries that are found on the content. This means that if you're indexing an NTFS share that only the DOMAIN\HR users can access, you don't have to worry about the search engine yielding those documents as search results when searches are done by the DOMAIN\Marketing folks. This holds true for SharePoint content which also exposes ACLs to the SharePoint indexer.

If you're completely new to search, consider reading the search chapters in Bill English's Microsoft Office SharePoint Server 2007 Administrator's Companion. While it's not specifically geared to search, it'll speak to search in general and all the topics directly related to search. It's probably the most complete SharePoint reference out there. I like to think of it as the MOSS Bible for all things general and out of the box. If you're looking for a more in depth title more specifically targeted to search, consider Inside the Index and Search Engines: Microsoft Office SharePoint Server 2007 (PRO-Developer).

Fire It Up

If you haven't done so already, consider creating a portal site for your enterprise. After that, create a Search Center somewhere off that root site.

All that's left to do is open up the Shared Service Provider for your farm and start adding content sources. After you perform a full crawl, a myriad of documents that used to be extremely hard to find (or at least required Sherpa like navigation knowledge) will start to show up as search results for anyone who had access in the first place. Start small if you have a tonne of content to index and aren't currently in a large farm set up, but have a little faith in the product too. Even a stand alone installation can easily maintain an index for thousands of documents, and when you need to scale your farm out, rest assured that it's very possible.

If you're part of a huge organization then search becomes a more interesting topic in the sense that indexing millions of documents and setting up the security around all this content is a big job. But the point of this post is to make you more aware of SharePoint search and took at it as a relatively easy value add. Take comfort in the fact that this out of the box search feature (MOSS Standard/Enterprise) will do things that even $10,000 Google Mini's won't. You owe it to yourself to at least start playing with it.

In future posts we'll get into Managed and Crawled Properties, search Scopes, Content Sources etc..., the purpose of this is simply to get you enthused about search. There's no replacement for good planning and a solid information taxonomy, but when information is already scattered to the winds nothing helps round up lost documents like a good search engine.

Good Luck,
Tyler

STSADM Restore Errors With SQL SERVER Express

2009-01-15T17:43:00.000-08:00

But There's Tonnes of Disk Space!

The other day I was migrating a site collection from one machine to another. I had a copy of the source site collection and was ready to restore it on this new machine that another developer had prepped. I created a SharePoint web application at the destination and then proceeded to run the restore command:

stsadm -o restore -url http://someurl -filename c:\file.bak -overwrite

After a while I ended up getting some disgruntled feedback from the stsadm tool:

The site collection could not be restored. If this problem persists, please make sure the content databases are available and have sufficient free space.

Which was confusing, I made sure there was tonnes of disk space (20 GB free for the restore of a 3 GB site collection) and tried again...it died with the same error. After poking around I noticed something odd after running the following query in the Management Studio:

SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

As it turns out, the database server I was trying to restore the site collection into was running SQL Server Express Edition which has a file size cap of 4 GB. When this "farm" was set up, it was done so with a Basic install which by defaul installs SQL Express and inherits it's limitations. The site collection I was trying to restore was breaking the 4 GB limit and as a result the stsadm tool thought there wasn't any disk space available, when really there was a bunch to be had..

Upgrading SQL Server Express to Developer/Standard/Workstation/Enterprise

As a result I had to upgrade the OFFICESERVERS instance of SQL Server running on that machine from Express to Developer (you can upgrade it to Developer/Standard/Workstation/Enterprise, whatever you have in terms of media/licensing).

This ends up being pretty easy, you simply find the media/installer and run the setup.exe that you normally would but with an added argument of SKUUPGRADE=1.

setup.exe SKUUPGRADE=1

When the install wizard gives you get a chance, select the OfficeServers instance and upgrade the database engine. If you want a more detailed walkthrough, one is available here. When you're done, run the same SQL query above and the product edition should no longer be Express.

After that small adventure, the restore worked like a charm.

Hope that helps someone.

Best,
Tyler

Big Fat Content Deployment Jobs

2009-01-13T00:24:00.001-08:00

The Blessings and Woes of Content Deployment

If you've spent a lot of time dealing with Incremental Content Deployment Jobs, then you've most likely either developed the patience of a saint, or a taste for 80 proof liquor.

This particular feature has worked, and then not worked over a series of SharePoint patches. For example, it didn't work under RTM for many sites, then a hot fix corrected the behavior. It was then broken again for most web applications under service pack one, which was followed by a fix in the Infrastructure Update.

Because this particular feature has dealt the world so much hair loss, a lot of clients I've met with have resorted to using full content deployment jobs to get their content from one site collection to another.

The key difference between the two is that an incremental deployment job will deploy only changes (new records, updates, and deletes). A full content deployment job will deploy everything in the site/branch/site collection that you select (that is, it will deploy a copy of the current version, not the version history as well). If you select a site, you're about to get another copy of every asset in that site. If you select the entire site collection, there's even more content that's about to get duplicated.

Hold Off On That Full Content Deployment

The real problem with resorting to full content deployments is that if you're prone to using them too often your content database size can go through the roof. For those who normally deal with SharePoint capacity and planning, this can be a nightmare. The last thing you want is bloated a content database with duplicate copies of 40 MB PowerPoint presentations coupled with the HR departments prized PDF collection.

A particular client we work with ended up using full content deployments for sub sites about 6 months ago because they couldn't get incremental content deployments to work. Before they knew it their destination site collection was 16 times the size of the source site collection. Luckily this particular content database was pretty lean (about 150 MB, and so the destination only ended up being ~2.5 GB). Imagine if they had a source site collection that was around 50 GB (which is very common for many intranet site collections), they'd be looking at a destination site collection which would be ~ 0.8 TB, a near unmanageable (or at least painful) amount of data for most small to medium IT shops.

No one really wins when it comes to regular full content deployments. Even if there's quotas on the site collection the inherent duplication still steals space that users could otherwise use to store useful data. If patching it isn't something you feel you can easily do yourself consider contacting MS SharePoint Support, they're not half bad, and they're pretty cheap ($250 last time I checked).

If you're not already convinced, here's a couple screen caps of data pulled from Red Gate SQL Data Compare detailing the differences on just a small site. The first cap is a before/after comparison of a sub site being deployed using an incremental job after a small change as taken place (content edit).

The second screen cap is of the same site before/after a full content deployment job targeting the same sub site.

Record Difference Before and After an Incremental Content Deployment

Record Difference Before and After a Full Content Deployment

These results will of course dramatically change with the site of your site collection and what you're doing a full content deployment on. The point is just to offer a reminder that the unnatural growth of the content database just may not be worth it in the long run. If you're currently wed to full content deployments I'd suggest either getting a divorce ASAP, or investing in some sizable disks to help manage the data explosion.

Buying more disks,
Tyler

Don't Rule Out MS Support For Help

2009-01-07T21:39:00.001-08:00

Someone Call Redmond

SharePoint 2007 has been out for over two years now. Even though I'm a fan of the product, I'd be the last one to tell you that there haven't been glaring bugs with some of the more prominent features. The good news is that over time, hot fix by hot fix, these are slowly getting resolved.

Depending on what you've been using SharePoint for, your mileage may have varied significantly. Until a couple days ago, all of the solutions and workarounds I'd ever applied to the product had come from forums, blogs, white papers, and books. Last week we ran into a problem that we’d flailed at for long enough and decided to enlist the help of Microsoft Support. This is a recount of that experience.

Issues with Content Deployment

We were running MOSS 2007 SP1 and were having problems with incremental content deployments. We were considering deploying the SharePoint Infrastructure Update which supposedly addresses the issue. We were reluctant to deploy the updated because of the related downtime our farm would incur (this particular farm is public facing and gets tens of thousands of visitors a day). If possible we wanted to find another way.

So we decided we’d push the problem onto The Soft and let MS Support deal with it. $250 later, a support ticket was opened. Contrary to the complaints I’d found online there was little to no wait, and within a couple business days someone from MS Support was calling me every morning asking me when we could start working on the ticket.

Although it was probably a different story when the product first launched, it doesn’t seem like there are Black Friday style wait lists for support these days. We also read this post by Eric to help us properly characterize the problem for MS Support. It’s of note that even though we provided a fair amount of detail in the opening emails, I ended up repeating the details back to the support agent anyways.

The troubleshoot went surprisingly well. The support engineer was attentive to our concerns and very thorough in her diagnosis. Although she had a very distinct accent, she was on our time zone (PST) and had exceptional product knowledge. What’s even better is they were willing to work around my extremely varied schedule and were relatively high touch. Someone from the support team phoned and emailed me at least every day, even if I forgot to return their emails. At times I felt like they wanted to fix our issues more than we did. All in all I’d give the experience an 8/10 and recommend it to anyone who feels like they’re grasping at straws.

The money ($250) is relatively cheap compared to your time, and you’ll get a chance to ask a ton of product related questions. What’s even better is that if your problem ends up being fixed by a patch (hot fix/update/service pack) or is related to a product deficiency (you use a work around to resolve it) there’s a strong chance that you’ll be refunded the $250. Ultimately the decision is supposed to be up to the support engineer’s manager. All this is from the mouth of a support engineer, so give it the same credence as you would most hearsay.

I guess the point of all this is to not rule out MS Support when it comes to SharePoint issues. By all means exhaust your regular avenues first, but don’t be afraid to ask for help from Microsoft Support when you start to feel all alone. The process requires a lot of patience, but depending on the problem it may be the cheapest way to fix the problem. Ideally having good support is one of the reasons your company decided to go with a well known vendor in the first place.

One Less Problem,
Tyler

Passion is Always Recession Proof

2008-12-30T21:10:00.001-08:00

A Good Recession?

A couple of weeks ago, in the midst of more economy doom and gloom, our CEO gave a company address speaking to the benefits of a recession. He then went on to make the point that recessions can actually be good for some companies.

In short, recessions reward good management. Companies who've had the discipline to not overextended themselves financially, stay within their circle of expertise, and are passionate about what they do, are excellently positioned to prosper in a recession.

As the competition starts to thicken and there are fewer customers to go around, mediocre companies inevitably fall apart or shrink substantially. Meanwhile great companies relish the opportunity to compete against their counterparts.

A recession separates the pretenders from the contenders. Firms who have built strong relationships with their clients and continue to deliver exceptional value, often have little to fear from a recession.

If anything, a recession helps good companies by weeding out all the chaff polluting the business space. So if a recession can help great companies, how about great developers?

For The Love = For The Win

Let's face it, a lot of IT personal (including developers) aren't exactly passionate about what they do. Somehow somewhere, a lackluster wave of hires stumbled into the IT workplace.

Maybe it was during the dot com days when insane salaries were lobbed at anyone who could work a keyboard. Maybe a career advisor or two read an aptitude test upside down. Either way I'm sure at some point you've bumped into a programmer who isn't passionate about programming.

Not that being apathetic towards your job is a sin per se, but it if you're not passionate about something, you'll never be exceptional at it. And if the most you're destined for is average, then both you and your team will be having perpetual dates with mediocrity.

People who are passionate about programming, would be doing it whether they're getting paid to or not. They're a lot more likely to be reading technical blogs, show up at user groups, and help a team grow deeper technically. And get this, they'll actually enjoy doing it!

So here's the good news about recessions for developers.

The Cull

Don't get it twisted, a lot of good people get let go during recessions. Huge swaths get cut through IT groups, and a lot of great staff ends up on the wrong side of the red line.

But just like great companies, great developers don't have a hard time finding work. Their managers are more than happy to write them glowing letters of reference and refer them on to other firms that can make use of their skills. The same passion that has kept them up to speed with industry changes will help them shine during the interview process.

It's not just good staff that gets laid off, bad hires also get let go too, and this helps development teams tremendously. Managers who once had the luxury of keeping bad hires are finally forced to take a hard look at their staff. The end result is often a more proficient and passionate team of developers.

Hiring is also a lot more productive during a recession, not only are there more great candidates to choose from, but bad hires are a lot more likely to occur when there's too much work available, as oppose to when there's too little. The net result is that teams tend to get more competent as only good developers get filtered back in.

Lastly, people who are simply stumbling around an industry are encouraged to take a moment to take a hard look at their current career path and decide if it's aligned with their strengths and interests. This isn't just good for the industry, it's essential for individuals to find something that they're truly passionate about and that they can find lasting success in. Keeping distracted people in an industry when their strengths really lie elsewhere not only erodes the craft, it steals valuable time from them that they could use to find something that they're exceptional at. Time they could otherwise ply at a trade more closely coupled to their natural strengths. When a bad hire is made, it's not only unfair to the company, it's wasteful of the employee's time. Recessions help hit the reset button.

Another Take

So depending on how optimistic you're feeling you could say that recessions are pretty good for you the developer. They help people find better jobs, stop bad hires, move people to industries where they can find real success, and reward those who truly belong in a space. Ideally it'll help improve your team, your company and your industry. If you're lucky you just might hit the trifecta.

Just a thought,
Tyler

Put Down That Ajax Control Toolkit!

2008-12-22T21:58:00.001-08:00

Out with the Old

Don’t get me wrong, I like old things. Constructs like the abacus, scythes, and slide rules will always be remembered by those unfortunate enough to have used them. But to continue using dated tools in today’s ever changing environment is like running cavalry against a line of tanks. As you continue to post mortem projects you should also be auditing your tool belt. This post is about me retiring not only a single tool, but an entire toolkit.

The AJAX Control Toolkit

The AJAX Control Toolkit was released under such a name in January 23^rd, 2007, prior this namesake it was known as ATLAS. This toolkit is built on top of the AJAX Extensions and includes a series of controls to help ASP.NET developers take their user experience to the next level. The toolkit boasts many controls like animation extenders, better validators, modal windows, and widgets like accordions and drag panels. A lot of developers (including me) jumped at these controls and were all too happy to include the dependencies in their projects in exchange for the near free functionality. I should know, I was one of them.

The problem with these controls isn’t that they suck; the problem is that they’re not the best. While Microsoft was developing these controls, other client side efforts were underway. You’ve probably heard of these other libraries, some of them being jQuery, prototype, and mootools. I’m personally ditching the toolkit in exchange for jQuery.

jQuery

If you haven’t heard of jQuery by now, I’d kindly suggest that you get out more often. I personally think of jQuery as the Firefox of JavaScript libraries. A lot of jQuery’s strengths come from a litany of professional grade plug-ins that have been contributed and then vetted by the jQuery community. It’s this same community, large in number and energized by a passion for better UIs that makes jQuery such an impressive offering. This is truly a technology that has a bright roadmap.

I decided to ditch the ACT (Ajax Control Toolkit) for jQuery for the following reasons:

Dependencies: The ACT has server side dependencies (both the System.Web.Extensions.dll and AjaxControlToolKit.dll need to be present), jQuery has no server side dependencies besides the emitted JavaScript. This same JavaScript is also smaller than that emitted by the ACT.

Controls: Most likely, in the time that it took you to read this blog post, another jQuery plug-in was submitted to the community. There are a lot of talented JavaScript developers that are using this framework to make some truly impressive plug-ins. I urge you to check out the plug-ins page, it more than dwarfs the ACT offering.

Collaboration: Developers may make it functional, but designers make it usable, and sadly the latter almost always trumps the former. Designers need to be able to mock out UIs and style these controls. The more they get involved the less you have to do when it comes to presentation. For designers to develop style sheets, they often need to get access to the control.

For the ACT this often means they need to be running Windows (which designers seldom do), install Visual Studio, and upgrade to either .NET v2.0 + System.Web.Extensions, or .NET v3.0 (sound tedious yet)?

Or they could just work with jQuery UI, a site that helps designers familiarize themselves with the markup and JavaScript needed to produce some great interfaces.

The ACT doesn't offer natural middle ground between developers and designers; it also doesn’t make for a shared ownership of the UI. Designers should be involved all the way through the project pipeline, not simply handing off a style sheet to a developer.

Passing the style sheet to a developer who didn't create the layout and isn't responsible for it, is likely to fast track your web application to ugly ville.

Worse yet, it’s likely the developer will change the style sheet if s/he can’t get markup to be emitted exactly like the designer was planning for (such is common when working with the ACT). This makes it increasingly awkward for the designer to maintain the style sheet as it moves to QA and then production. The ACT just isn't very designer friendly.

Don't Take My Word For It

Learning jQuery not only gives you access to a tonne of great controls; it starts you down the process of learning a DOM manipulation framework that is truly platform independent. If you’re concerned about support from a big company, Microsoft recently decided to include jQuery with Visual Studio as of September 2008. Consider using jQuery in SharePoint to spruce up a dull UI, or to make ASP.NET MVC more palatable. I think it’s time to get on the bus, this is one that you don’t want to miss.

Best,
Tyler

Advanced Basics: What Causes a JIT?

2008-12-19T00:32:00.001-08:00

Kind Of A Black Box

For the longest time ASP.NET Just In Time compiling has always been a pretty gray matter for me. Word on the street has it, that some things you'll do against a web site cause the site to be re-JIT'd. Other actions will simply cause the application to unload. But which actions precipitate which results?

Just In Time Compiling

As you're probably aware, the code that you put in an ASP.NET web site isn't the actual code that runs. All those assemblies, web pages, user controls etc... all need to be Just In Time Compiled before they can actually be used. Visual Studio creates MSIL (Microsoft Intermediate Language) a bytecode that is highly portable. Before you can run it a .NET runtime, it needs to be compiled again into native code. Native code is specifically targeted to the machine it's going to be running on and is a lot more efficient. You can do this yourself (assuming you don't want your code to Just In Time compiled) by using either the ASP.NET Precompilation tool or Ngen.exe (Native Image Generator) another tool that comes with the .NET SDK.

All that's required for MSIL to be JIT'd is that the destination machine have a compiler that's capable of optimizing your MSIL for the given environment. A common example of this is when a developer builds an application on a 32-bit machine and then goes and deploys that MSIL on to some 64-bit machine. At runtime the MSIL will get natively compiled into 64-bit assemblies by a compiler that's sensitive to the needs of the destination 64-bit machine.

The native compiler may also do things like (from Wikipedia):

Optimize to the targeted CPU and the operating system model where the application runs. For example JIT can choose SSE2 CPU instructions when it detects that the CPU supports them. With a static compiler one must write two versions of the code, possibly using inline assemblies.
The system is able to collect statistics about how the program is actually running in the environment it is in, and it can rearrange and recompile for optimum performance. However, some static compilers can also take profile information as input.
The system can do global code optimizations (e.g. inlining of library functions) without losing the advantages of dynamic linking and without the overheads inherent to static compilers and linkers. Specifically, when doing global inline substitutions, a static compiler must insert run-time checks and ensure that a virtual call would occur if the actual class of the object overrides the inlined method.
Although this is possible with statically compiled garbage collected languages, a bytecode system can more easily rearrange memory for better cache utilization.

That's pretty much the just of it. Just In Time compiling adds a lot of flexibility to .NET development. Even though you've built a very generic application (can run anywhere with a CLR), what really get's run is a highly optimized version of that code that is targeted to the destination machine's architecture, operating system, etc...

How Does This Happen (in ASP.NET)

When you first visit a web application it needs to at least JIT the important stuff. JIT'ing starts with what's called top-level items. The following items are compiled first (pulled from here).

Item	Description
App_GlobalResources	The application's global resources are compiled and a resource assembly is built. Any assemblies in the application's Bin folder are linked to the resource assembly.
App_WebResources	Proxy types for Web services are created and compiled. The resulting Web references assembly is linked to the resource assembly if it exists.
Profile properties defined in the Web.config file	If profile properties are defined in the application's Web.config file, an assembly is generated that contains a profile object.
App_Code	Source code files are built and one or more assemblies are created. All code assemblies and the profile assembly are linked to the resources and Web references assemblies if any.
Global.asax	The application object is compiled and linked to all of the previously generated assemblies.

After the application's top level items have been compiled, ASP.NET compiles folders, pages, and other items as needed. The following table describes the order in which ASP.NET folders and items are compiled.

Item	Description
App_LocalResources	If the folder containing the requested item contains an App_LocalResources folder, the contents of the local resources folder are compiled and linked to the global resources assembly.
Individual Web pages (.aspx files), user controls (.ascx files), HTTP handlers (.ashx files), and HTTP modules (.asmx files)	Compiled as needed and linked to the local resources assembly and the top-level assemblies.
Themes, master pages, other source files	Skin files for individual themes, master pages, and other source code files referenced by pages are compiled when the referencing page is compiled.

Conditions

The following conditions will cause your application to be re-JIT'd. If you modify any of the top level items, any assemblies that reference the top level items will be recompiled as well.

Adding, modifying, or deleting assemblies from the application's Bin folder.
Adding, modifying, or deleting localization resources from the App_GlobalResources or App_LocalResources folders.
Adding, modifying, or deleting the application's Global.asax file.
Adding, modifying, or deleting source code files in the App_Code directory.
Adding, modifying, or deleting Profile configuration (in the web.config).
Adding, modifying, or deleting Web service references in the App_WebReferences directory.
Adding, modifying, or deleting the application's Web.config file.

Whenever one of the above conditions is met the old application domain is "drain stopped", which means that new requests are allowed to finish executing and once they're finished the Application Domain hosting those assemblies unloads. As soon as the recompile is finished a new Application Domain starts up with the new code and starts to handle all new requests.

Exodus

Believe it or not, having this kind of knowledge somewhere on hand will actually help you when you need it the most...when you're troubleshooting. This kind of stuff doesn't need to be kept at the tip of your tongue, but it should be at least semi-salvageable from the recesses of memory. The worst kind of trouble is the kind that doesn't make any sense. Having sophisticated tools that do a lot of leg work for you can also put bullet holes in your feet if you haven't read the instruction manual.

Hope that provided some value. I swear this stuff comes up more than you'd think.

Best,
Tyler

Application Domains Revisited

2008-12-13T14:33:00.001-08:00

I Should Know This

There's a lot of things I've taken the time to learn formally (ie. go to school, take a cert) which escape me when they would actually come in handy. In fact it becomes even more frustrating when we end up figuring out what happened...and it's something I already (should) know.

This once again became wildly apparent this week while I was troubleshooting some random web app. As it turns out, poor memory is the gift that keeps on giving, and my own memory is definitely feeling the festive nature of the holidays. In a very small nutshell, I forgot the following.

Every application you set up in a web site (including the root application) ends up being in its own application domain. It'll have its own set of assemblies, get JIT'd independently, and of course be completely isolated from the the contents of other application domains (Session, Statics, Cache, etc... won't be shared.).

Refresher

Process: Contains the code and data of a program. Also contains at least one thread. Represents a boundary that stops it from wreaking havoc on other processes and requires special techniques to communicate across. Runs in a security context which dictates what it can and can't do. If it's running some .NET code then it contains one or more application domains.

An example of a process might be notepad.exe, or if we're talking about IIS aspnet_wp.exe (Win XP, Win 2000) or w3wp.exe (Windows 2003).

Application Domain: Are more efficient than processes. One process can load the .NET framework once and host many application domains. Each app domain might load its own assemblies (or even it's own child application domains), but will not have to reload the overhead of the .NET framework. Since each application domain runs within the .NET framework, it benefits from the features there of (things like Code Access Security, Managed Memory, JIT Compiling, etc...).

Application Domains also represent a boundary that you'll need special techniques to communicate across. Since more than one can be hosted in an process, it's common for the .NET Framework to load up and unload application domains at runtime. This is what happens when you recycle an IIS Application Pool, the application domains within it are unloaded and new app domains are created to service new requests.

In ASP.NET an Application Domain is created for every virtual directory that you configure and as application in IIS. Each can have it's own bin folder, web.config and IIS Application Pool.

It's worth mentioning that even though there's Application Domain boundary separation between applications, they can still inherit web.config settings from each other (if one is in a parent folder).

A high level view of Processes, the .NET Runtime, Application Domains, and the assemblies they in turn load might look like the following picture.

Application Pools (IIS 6 and above): Host one or more applications (Application Domains if we're talking about .NET code). If an application pool is running in worker process isolation mode, it can also spread those applications over one or more processes (a web garden)! They allow you a great deal of flexibility when it comes to deciding how much isolation you want to give your applications. You can use application pools to provide process level isolation for each application (each application can have its own w3wp.exe [or more if you web garden]). Application Pools can also combine multiple applications in a single application pool (saves resources).

Below, an application pool (".NET v2.0") spreading it's work over multiple processes.

And We're Back

Every now and then I get a little scared of the complexity that will be in development environments 20 years from now. Yes, the stuff being written will be extremely cool, and the tools that get used will no doubt be just as impressive. That being said, there will be a a lot of complexity in play in the future and a tonne to be aware of.

With .NET v4.0 in the works, a lot of focus seems to be on parallel computing (ie. PLINQ) I can't imagine the computing world is about to get much simpler.

Keep your seat belt on.

Best,
Tyler

The SharePoint Certification Gauntlet

2008-12-07T15:35:00.000-08:00

Finish Line

This week I finally finished the last of the SharePoint certifications. They include:

MCTS: 70-542 (MOSS Application Development)
MCTS: 70-541 (WSS Application Development)
MCTS: 70-630 (MOSS Configuration)
MCTS: 70-631 (WSS Configuration)

Doing all four in a single year was a little more time consuming than I thought it would be, and I'm glad it's finally over.

The Value

While certifications themselves don't really get you deep in a subject matter (at least not that I've noticed), they do give you a really good idea of the surface area. At that point you're in a good position to explore the topics on you own and get deep in a few.

Here's a more concrete example. A cert in ASP.NET won't get you deep in HTTPModules, but it will tell you what they are, and what you might use them for. The next time you hit a problem that involves say URL Rewriting, you've still got some learning to do, but you're a lot less likely to leave the reservation completely and come up with something home brewed and semi exotic.

To me that's one of the biggest values of certifications, they try to clearly define the scope of the technology. After a cert you should at least know what you don't know about some tech stack. At that point you're a lot less likely to write a bunch of code to solve a problem that your tech stack is naturally geared towards solving.

Rounding The Edges

Of course each of the four SharePoint exams have a slightly different focus. It's also worth mentioning that two of them don't really have anything to do with writing code, they're all about configuration. I thought these were important because of SharePoint's farm footprint. The platform is a lot more than a single web server/database. It's really a series of services working together on many different machines. When all these services act in concert, the resulting ballet provides a pretty decent platform for collaboration.

Because there's so much in play (and so much complexity), it seemed just as important to me to learn the infrastructure nuts and bolts as it was to learn the various APIs.

This sentiment was recently corroborated by the SharePoint Master program. The program involves the same four certifications. I'm pretty sure the content authors of the program are on the same page, that you can't be a "SharePoint Master" unless you have a holistic knowledge of both infrastructure and application development.

Go For It

For those wondering how to go about getting a certification I would encourage them to do so. The experience alone will tell you if you feel it was worth its salt. The typical certification route looks like this.

Find a technology that interests you and a certification path on the Microsoft Learning web site.
Read the about the test requirements. This may involve one or more exams that each have a preparation guide. Be sure to follow the preparation guide.
Study up. This might involve some online learning, some books or just spending a lot of time on the MSDN.
Take a Practice Test. Real exams cost $125 and a couple hours of your time. It's doubtful you'd want to want to write the real exam more than once. Most exams require at least 70% as a passing grade, I've written exams that required as much as 80% to pass. Make sure you're ready before you schedule an exam. You can often find free practice exams online, failing that just pay the $70 to a practice test provider (MeasureUp, Transcender, TestKing, etc...). The idea is to make sure you're ready before scheduling the real exam.
Schedule an exam. For Microsoft exams you'll most likely end up going through Prometric or Pearson Vue. Their web site will book you an appointment with a testing provider (most often some IT college or learning center). Exams cost $125 USD and usually need to be scheduled at least 48 hours prior to writing.
Show up and ace the exam!

Everyone learns at a different pace, but expect to spend at least fifteen hours going over material and practice exams for your first certification. After writing a couple exams you'll notice your preparation start to decrease as you start to streamline the process.

After finishing a certification you'll be given an MCP ID (if you don't already have one), a welcome kit, a certificate, and use of a certification logo (below) for business cards and such. I don't personally make use of the last two, but they may help demonstrate to your boss that your passionate about technology, or convince women you're able to commit to something.

Remember that certs are just a part of your learning continuum, supplement them where need be. They're nice to have, but by no means the finish line.

Best,
Tyler

Windows XP Unzip Errors With SharpZipLib

2008-12-02T17:41:00.000-08:00

Great Utility

SharpZipLib is a great little library that helps you programmatically compress/decompress zip archives. If your needs aren't too exotic (ie. you need to programmatically zip/unzip a series of files/folders) this could very well be your ticket.

You've probably heard of it before (it's been around for quite a while), some sample usage might look like this (compresses a file).

using (ZipOutputStream zipStream = new ZipOutputStream(File.Create(zipFilePath)))
{
 //Compression level 0-9 (9 is highest)
 zipStream.SetLevel(GetCompressionLevel());

 //Add an entry to our zip file
 ZipEntry entry = new ZipEntry(Path.GetFileName(sourceFilePath));
 entry.DateTime = DateTime.Now;
 zipStream.PutNextEntry(entry);

 byte[] buffer = new byte[4096];
 int byteCount = 0;

 using (FileStream inputStream = File.OpenRead(sourceFilePath))
 {
     byteCount = inputStream.Read(buffer, 0, buffer.Length);
     while (byteCount > 0)
     {
         zipStream.Write(buffer, 0, byteCount);
         byteCount = inputStream.Read(buffer, 0, buffer.Length);
     }
 }
}

This is pretty normal usage. It adds a zip entry to an archive and creates said archive. What might confuse you though is the error you'll get if you try to to unpack the archive using a legacy unzip tool (ie. the stock Windows XP decompression tool).

The stock Windows XP extraction tool won't be able to unpack the archive and will complain about the archive being "invalid or corrupted" if you try to extract the archive contents.

Zip64 Extensions

What's happening here is that the utility is enabling Zip64 extensions for the archive, and some older utilities can't read Zip64 Extensions. You could simply turn off Zip64 Extensions, but this will cause problems when you start adding files larger than 4GB to your archive.

A better solution is to make a mild tweak to the way we add files to the archive. By specifying the size of the file we're adding, the ZipOutputStream can decide whether or not to use the Zip64 extensions. If we don't need them then they'll be turned off automatically. The mild tweak below fixes the error from the above code:

...
ZipEntry entry = new ZipEntry(Path.GetFileName(sourceFilePath));
entry.DateTime = DateTime.Now;
/* By specifying a size, SharpZipLib will turn on/off UseZip64 based on the file sizes. If Zip64 is ON
* some legacy zip utilities (ie. Windows XP) who can't read Zip64 will be unable to unpack the archive.
* If Zip64 is OFF, zip archives will be unable to support files larger than 4GB. */
entry.Size = new FileInfo(sourceFilePath).Length;
zipStream.PutNextEntry(entry);
...

Hope that helps someone. That error drove me crazy for a while and I didn't find much via Google.

Best,
Tyler

Does SharePoint Lend To High Availability?

2008-12-01T08:34:00.001-08:00

All This Hardware And No Uptime

SharePoint is pretty heavy. I often think of it as an 800 pound gorilla who stopped exercising and let itself slide. To handle all the services that run within a farm and provide decent response time to users, a reasonable amount of hardware usually gets provisioned to pick up the slack.

I'm talking about real iron here. Large farms featuring clustered SQL Servers, redundant application servers, and a series of web front ends balanced with either a network load balancer or a Microsoft NLB cluster.

One might look at all this gear and think that as a result, the farm is almost guaranteed to enjoy some pretty high availability right? Well I guess that depends on what you call high availability.

A Desire for High Availability

\	Total downtime (HH:MM:SS)
Availability	per day	per month	per year
99.999%	00:00:00.4	00:00:26	00:05:15
99.99%	00:00:08	00:04:22	00:52:35
99.9%	00:01:26	00:43:49	08:45:56
99%	00:14:23	07:18:17	87:39:29

For clients that are running SharePoint internally, uptime is probably important but not a huge priority. Clients that use SharePoint for internet facing applications are another matter. These users are far more likely to use SharePoint to buttress e-commerce offerings or brand efforts. These businesses usually want high availability and may even ask for four to five nines of uptime (99.99%-99.999% uptime). Five nines (sometimes called the holy grail of uptime), equates to being down no more than 5 minutes 15 seconds a year. It's a bold proposition and requires a great deal of planning and forethought. Still, if it's doable, all this hardware should set you in the right direction right?

Heavy Patches, And Lots of Them

The biggest hurdle I've had with providing high availability to clients with SharePoint has come from the patching procedures issued from Microsoft. Normally when updating applications/machines it's possible to update one machine at a time, using your load balancer to shelter this machine from production. Once the machine has has been updated you can bring it back into production and start updating one of it's siblings. With SharePoint this process gets a little more complicated. Here are a couple of the reasons:

There's no uninstall/rollback for most SharePoint updates (your best bet for uninstall is a machine level backup).
The recommended install procedure dictates that you stop all IIS instances for Web Front Ends. This makes it difficult to continue to provide service or at the very least hold up a stall/maintenance page.
The recommended install procedure asks that you upgrade all machines on the farm at the same time.
There's usually at least one machine in the farm that rejects the upgrade and needs to be troubleshot individually. For me these have often resulted in removing the machine from the farm, upgrading it, and then adding it back to the farm. This usually adds to server downtime, especially if the server was serving a key role (ie: SSP host or the machine that hosts the Central Administration web site).

Assuming you manage to make it through all the above without a lot of downtime, how many times a year do you think you might be able to do it and still maintain a reasonable downtime SLA? Before you answer that, consider all the updates that have come down the pipe for WSS since it's RTM (it's SharePoint 2007 remember). This is also just a list of updates for WSS, there's a whole other table for MOSS (although most of the dates and versions coincide).

Update Name	Version (owssrv.dll)	Date
Windows SharePoint Services v3.0	12.0.4518.1016 (RTM)	November 2006
October public update (2007)	12.0.6039.5000	October 2007
Service Pack 1	12.0.6219.1000	December 2007
Post-Service Pack 1 rollup	12.0.6300.5000	March 2008
Infrastructure Update	12.0.6320.5000	July 2008
August Cumulative Update	12.0.6327.5000	August 2008
October Cumulative Update	12.0.6331.5000	October 2008

Don't get me wrong, updates are good. In fact, I like it when Microsoft fixes things, especially when the clients who have purchased MOSS have already paid potentially millions in licensing fees. I just wished these updates which happen many times a year AND provide critical fixes to expected functionality had better upgrade strategies.

Do SharePoint updates and the way in which SharePoint farms are upgraded make high availability a pipe dream? Does all that hardware do nothing except help the farm scale out?

A Little Transparency

In fact all I'm really looking for from these updates is a little transparency. I'd be thrilled if I could get a little more detail as to what's going on underneath the hood and what to do when the error log starts filling up.

I've yet to see a really good troubleshooting strategy or even deployment strategy that gives you good odds of limiting downtime when it comes time to roll out these upgrades.

We have a ticket open with MS support to take up this issue. The wait for SharePoint related issues is still pretty long, but rest assured should I come up with one or find a good resource for these kinds of rollouts you'll know where you'll find it.

Best,
Tyler

Effortless Hot Cloning with VMWare Converter

2008-11-25T23:07:00.001-08:00

Pulling Teeth

It's an exceedingly common scenario these days to want to make a virtual machine out of some physical machine. In fact when it comes to reproducing an environment I can't think of a more thorough way (especially when there's multiple machines involved).

This often leaves someone tasked with finding a powerful, affordable, and flexible physical to virtual conversion utility. Now finding a P2V solution isn't really a challenging problem, but finding one that's easy to use is.

I would encourage anyone who's ever been frustrated with the P2V tools on the .VHD (Microsoft Virtual Server, Virtual PC) side to consider making a .VMDX instead (VMWare Workstation, Server, Fusion).

Novocain

The VMWare Converter is free, effortless to use, and makes hot clones (in addition to cold ones) with the same ease that Johnnie Cochran acquits his clients. With a hot clone you don't even need to shut down the OS or stop any services! You can perform a P2V transformation and experience zero down time...all for free (if you call being bound to a VM Ware product free). This utility is so handy it just may be the help you need to help you make a better vendor choice when it comes to virtualization...

Just walk through a wizard, click hit a few radio buttons, and you'll be creating a hot clone in no time.

Details

The VMWare converter will run on the following OS's:

Windows XP Professional
Windows 2003 Server
Windows 2000 Professional
Windows 2000 Server
Windows NT SP4+ (IE5 or higher required)

You're likely to get a much smoother ride with Windows XP and Windows 2003 as they both make use of the Volume Shadow Copy service, but supposedly you can make hot clones with the other OS's as well (I've only made hot clones on Windows XP and Windows 2003).

Like any task that could go horribly wrong, there's a couple of things you should check before selling yourself on hot clones, or at least placing any bets.

Don't just take my word for it, try the utility on some machine you don't like and create a clone (hot or cold). When you're done, just be sure post a YouTube video of you Office Space'ing the machine just like these guys did.

The clones should be made hot, the revenge should be served cold.

Note: Some employers take offense to employees taking their hardware into fields and striking company assets repeatedly with with sports equipment, regardless of whether or not a hot clone has been made.

Best,
Tyler

Sandboxes Grow Good Developers

2008-11-22T02:09:00.001-08:00

Uh, You're Testing That Where?

A while back one of our data analysts and a network admin stubbed out a sheltered domain within our LAN. The idea was that we would set up a sandboxed class C network where developers could test code and not worry about breaking anything on our public network.

While this might seem a little hardcore at first, remember that a lot of solutions these days are meant to be deployed to some reasonably complex IT environments (like this sample BizTalk topology on the right). As such these codes may be making calls to Active Directory, Exchange, sit underneath some type of proxy (ISA/Squid), and get deployed to some kind of web farm (think load balancer appliances, NLB clusters, etc...).

Needless to say these kinds of environments are dramatically different from what you find on your workstation. Depending on the type of application you're delivering and where it's getting deployed, simply testing on Windows XP and IIS 5 might not qualify as a realistic environment.

The Right Kind of Cheap

The good news is that with Virtual Server/PC being free (give or take a windows license), virtualization has never been cheaper or easier. We ended up using a bunch of hardware kicking around the office to provision these machines. Our little test subnet already has a wide variety of virtual instances running around in it, hosting everything from Exchange to AD, and even a full blown MOSS farm.

The bad news is that maintaining this virtual instance still costs time, and as you may have heard, time is money. This can be mitigated in part, by the ease of performing machine level backups on VMs, and other VM features like Undo Disks, Differencing Disks and saving state. Nevertheless it makes sense to budget some time to maintain this kind of environment, rarely does IT infrastructure maintain itself...

The Easy Value

Originally the thought was to set up some SharePoint farm and allow developers to stage their code in a farm environment prior to deploying at the clients. You'd be surprised at just how many deployment issues crop up just by deploying to a farm instead of a single machine.

The real value is that you give the solution a chance to break in the your farm first. I'd much rather start up a troubleshoot at my desk than in a client's server room.

As far as I'm concerned the farm has already paid for itself. It's already stopped me from deploying a MOSS update that I'm positive would have trashed one of our clients SharePoint farms. As some of you may know, there is no uninstall for most SharePoint Service Packs and Updates, you often have no choice but to continue forward...even if it's off a cliff.

By breaking things in our VM farm, not only do I have an easier time rolling back if I hit a wall, but I can also troubleshoot it with all the tools and brainpower that my fellow developers have to offer. The alternative is breaking it for the first time at some client location where I'm all alone and only have the tools I thought to bring with me. Some fights just shouldn't be fought alone (at least the first time around).

The Extra Value

This wasn't immediately apparent to me but a lot of developer benefit from this kind of sandbox for another reason. In addition to testing their code in a more complex environment, they actually get a chance to look under the hood and play with the many services that decorate typical IT environments. It's surprising how many of these services that your typical developer isn't familiar with (AD, DNS, ISA, IIS, MOM, etc...). Let me elaborate.

A ton of web developers have no idea how DNS works. This could also be said for HTTP, TCP and the web servers that host their applications.

This akin to a cab driver that knows how to drive, but doesn't know any traffic laws or anything about the car.

This isn't meant to be a roast for developers who haven't had the benefit of IT fundamentals. Some of these topics just aren't covered in a Computer Science degree. This is about enabling developers. Empowering them to discover what hidden gems exist in typical IT environments and how these existing IT assets can help deliver better solutions.

Trust me, there's a big difference between telling a developer what DNS is and simply having the guy create and A record and see that light switch on in his eyes. What's even better is letting him discover for himself what IT assets can do for him in a clean and controlled environment where mistakes are easy to rollback.

Imagine This

Don't get me wrong, there's always going to be a percentage of developers that are clueless when it comes to all things IT (starting with their workstation). That's where your job security comes from. But if you educate just a couple more bodies you'll start to notice the difference pretty quickly. There's a night and day difference between a dev who simply lobs a program over the wall and one who has a holistic understanding of the landing zone. These kinds of investments don't take long before they start to yield noticeable returns. There might even be a day where IT staff come to trust developers to not completely mangle production environments.

Just talking out loud...

Best,
Tyler

Creating a Custom CAS Policy File For SharePoint

2008-11-16T14:41:00.001-08:00

Some Reasons

Maybe you're tired of putting your SharePoint web application in full trust and running web applications with huge security surface areas. Or maybe you've become tired of GAC'ing assemblies just so that they can run in full trust. Heck, maybe you've even read this article and decided to finally start handling Code Access Security in a more elegant way.

All of the above are good reasons to make a custom Code Access Security policy file, and in the following section we're going to do just that. If you're curious as to why you might be doing this in the first place, it might be a good idea to peruse common code access security issues described here.

Creating A Custom Policy File

The intent of the custom policy we're about to make is to allow your code and only your code to run in full trust regardless of where it is (doesn't have to be in the GAC). All other code in the application will be running as WSS_Minimal which means it will have a reduced set of privileges (see table). For example, codes running in WSS_Minimal can't access the SharePoint object model. Your code however will be running in full trust and will be able to do whatever it wants.

This is often desirable since we have no idea what web parts information workers or administrators will lob into our SharePoint application. It's a little presumptuous to assume that they're all safe and won't do anything malicious. Granting them full trust allows them to do things like write to sensitive areas of the disk, access the registry, etc... Users with high privileges (ie. administrators) get duped in to running malicious code all the time, that's one of the reasons Code Access Security exists in the first place. If you're curious about what the difference between code running in full trust and code running in WSS_Minimal is, refer to the table in the link above. Onward.

First make a copy of C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\CONFIG\WSS_Minimal.config and put it in your application folder, call it WSS_Custom.config. If you want to increase the privileges of other code you can start with WSS_Medium, this will allow other code to access the SharePoint API without having security exceptions thrown.
Strong name your assembly. We're not going to put in the GAC, but it still needs to be strong named so that it can be uniquely identified by the WSS_Custom.config policy file.
Open up the Visual Studio Command Prompt (or go find sn.exe) and extract the public key from your .snk file into another file. (It's worth noting that the Public Key is not the same as the Public Key Token that you might get from your assembly using Reflector).
sn -p YourStongNameFile.snk PublicKeyOnly.snk
You should see the output "Public key written to PublicKeyOnly.snk".
Now we print out that public key with the following command:
sn -tp PublicKeyOnly.snk

Copy that insanely long stream of numbers (your assembly's public key). Add the following line to your WSS_Custom.config just below the <CodeGroup class="FirstMatchCodeGroup"...>'s <IMembershipCondition> element.

<CodeGroup class="FirstMatchCodeGroup" version="1" PermissionSetName="Nothing">
<IMembershipCondition class="AllMembershipCondition" version="1" />
<!--[Your Entry Here]-->
<CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust">
<IMembershipCondition class="StrongNameMembershipCondition" version="1" PublicKeyBlob="[Your Public Key]"></IMembershipCondition>
</CodeGroup>
<!--[End Addition]-->
...
</CodeGroup>

Now ensure that you're web.config uses your custom policy file. Add the following line right under <SecurityPolicy>:
```
<trustLevel name="WSS_Custom" policyFile="[path to wss_custom.config]" />
```
Finally change your web applications trust level to use your custom policy file. Ensure that the <trust/> element looks like below:
```
<trust level="WSS_Custom" originUrl="" />
```
That's it, you're done! You may need to restart IIS if you get an "Assembly <assemblyName> security permission grant set is incompatible".

I've posted a sample WSS_Custom.config if the above was confusing.

If you were getting security exceptions before they should be gone now, at least all those coming from your assembly. Other code will continue to run in WSS_Minimal (or WSS_Medium if you used that .config as a template) and may throw security exceptions should they try to access APIs that they don't have the privileges to do so.

This is a lot more preferable than two popular alternatives:

Having the entire web application run in full trust (which gives all assemblies full trust).
Putting your assemblies in the GAC just so that they can run in full trust.

Hope that helps.

Best,
Tyler

Empty/Blank IIS Manager When SharePoint Timer Throws Error

2008-11-13T14:49:00.001-08:00

Ugly Error

I've seen this error often enough that I think it deserves a brief entry. Essentially the symptoms are:

When you open up the Internet Information Services (IIS) Manager it takes a long time to load and when it finally does the console is blank. Running iisreset (Start->Run->iisreset) will temporarily fix the problem.
In the event viewer under Application you see error codes for 6398, 7076, and 6482. Specifically they look like:

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Timer
Event ID: 6398
Date: 11/13/2008
Time: 1:34:47 PM
User: N/A
Computer: [COMPUTER NAME]
Description:
The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob (ID [GUID]) threw an exception. More information is included below.
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 7076
Date: 11/13/2008
Time: 1:34:47 PM
User: N/A
Computer: [COMPUTER NAME]
Description:
An exception occurred while executing the Application Server Administration job.
Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Techinal Support Details:
System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Server stack trace:
at System.DirectoryServices.Interop.UnsafeNativeMethods.IAdsContainer.GetObject(String className, String relativeName)
[...]

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 6482
Date: 11/13/2008
Time: 1:34:47 PM
User: N/A
Computer: [COMPUTER NAME]
Description:
Application Server Administration job failed for service instance Microsoft.Office.Excel.Server.ExcelServerSharedWebServiceInstance ([GUID]).

Reason: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Techinal Support Details:
System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Server stack trace:
at System.DirectoryServices.Interop.UnsafeNativeMethods.IAdsContainer.GetObject(String className, String relativeName)
[...]

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

A Fix

Supposedly this behavior takes place when the Windows SharePoint Services Timer (OWSTimer.exe) has two threads that try to access IIS at the same time. There's currently a hot fix and description for this problem that patches IIS that. Ideally there'll be an IIS update that comes out in the future that isn't a hot fix. If you can bear it I would consider waiting.

Best,
Tyler