So, why are statics bad?

I’m a huge fan of automated testing. I started with unit tests, but have recently moved into the realm of behavior tests. Yet, for any automated testing, you have to have good architecture in your code to ensure that you can test in any environment.

Let me give the most basic example: the use of a logging facility. Take for example the following loggger that I see all too often:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

public static class LoggingHelper
{
    public static void LogMessage(string message, string area)
    {
        var logPath = Constants.LogPath;
        using (StreamWriter writer = System.IO.File.AppendText(logPath))
        {
            writer.WriteLine("INFO\t[" + area + "]" + message);
        }
    }

    public static void LogError(string message, string area)
    {
        var logPath = Constants.LogPath;
        using (StreamWriter writer = System.IO.File.AppendText(logPath))
        {
            writer.WriteLine("ERROR\t[" + area + "]" + message);
        }
    }

    public static void LogException(string message, string area, Exception exception)
    {
        var logPath = Constants.LogPath;
        using (StreamWriter writer = System.IO.File.AppendText(logPath))
        {
            writer.WriteLine("EXCEPTION\t[" + area + "]" + message + " - " + exception.Message);
            writer.WriteLine("Stack Trace" + exception.StackTrace);
        }
    }
}

Here are a few areas the above code block can improve:

• There’s no exception handling. At all.
• There’s no mutexes/resource locking. Thus, it’s possible for multiple threads to attempt to open the file at the same time. The first one in will succeed. All others will throw exceptions.
• It’s insanely duplicative. In fact, I was able to copy every line of LogMessage to LogError and only have to change 2 lines of code. The same goes for LogException, where I also added an additional line.
• What happens when I have another type of message I need to log? That means I must add another method - 6 of those lines likely the same as LogMessage.
• What happens when I need to change the location of the log file?
• What happens when I need to change the underlying facility for logging? Say, from files to databases?

The last two are key here. For the first, I can change from Constants.LogPath to something like ConfigurationManager.AppSettings["LogPath"], But what happens when the file is a network file system? Not accessible via standard UNC paths? You’re up a creek and this entire class needs to change. That’s pretty much the case of the second question.

You should assume that your underlying logging destination can and will change at any time. If you think that the simple solution to this is to simply change the LoggingHelper class, think again. Let’s take this further. You should assume that your destination will be different for multiple scenarios. In enterprise development, this is frequently the case. In a pre-integration environment, you may only want to log to the filesystem because it may be a single server. However, in integration and production you might choose to log to a database since you’re deploying to a farm of servers and writing to the filesystem makes diagnosing problems a true headache.

Back to the problem of automated testing, you don’t really want any logging. Why? What happens when your test runner doesn’t have access to the logging path? What happens when your test environment doesn’t have the path at all? What happens when you forget you have your test environment dumping massive amounts of logs to the filesystem only to go on a goose chase figuring out why your builds randomly started to fail due to the disk being full? In the case of unit testing, /dev/null should be your destination.

Let’s consider one last secenario: what happens if you want to log to multiple facilities at once?

In any of those cases, using a static class with static methods does not afford you any flexibility. If you want to expand to multiple logging facilities, you would be forced to create another static class. That class is now called in addition to this one. Yuck.

Here’s where I come back to the SOLID principles. Let’s look at how Static classes (and static methods) violate these principles.

• Open/Closed Principle - Classes should be open for extension, but closed for modification.
  • With statics, you cannot extend and override the behavior. You throw out the idea of polymorphism. This makes it very difficult impossible for me to inject different behaviors for whatever need I may have.
• Dependency Inversion Principle - Depend upon Abstractions. Do not depend upon concretions.
  • With statics, every class that leverages its facilities are now dependent upon a concretion. There is no way to pass around an abstraction of a static class or regular class with static methods. Your only way of doing this is with reflection and that’s a definite code smell.

In summary, because I cannot leverage the power of inheritance, polymorphism, or really any aspect of object oriented analysis and design with statics, it feels a lot like procedural programming. I suggest leaving this practice behind.

So, why are helpers bad?

My discussion with my colleagues yielded an interesting observation: everyone’s opinion of a helper was slightly different. One thought of it as a single data access layer. Another thought of it as a way of providing consistent authentication for data access. But others think of helpers as the giant “I don’t know where to stick this method” bucket. So, you end up with something like this (methods only, since that should convey what I’m getting at):

1
2
3
4
5
6
7
8
9
10
11
12

public static class Utils
{
    public static string CleanInput(string input);

    public static void FlushCacheIndex();

    public static bool CheckIfSystemIsHealthy();

    public static int GetCountOfActiveUsers();

    public static MyModel CreateModelFromInput(string name, string email, int age, DateTime birthday);
}

These types of helpers are a maintainability nightmare. For starters, if you haven’t read my section above on why statics are bad, step back and read it. Both SOLID principles are still violated. Yet, there are also a couple of others that are violated.

• Single Responsibility Principle - A class should have only one reason to change.
  • It’s clear in the case above that there are multiple reasons this class exists. However, there are other helper classes (such as StringUtils which are not as clear. However, I’m still a firm believer that the Open/Closed Principle dictates whether to add yet another method to an existing class, or create a separate class. I’m a huge fan of small classes as their much easier to test and maintain.
• Interface Segregation Principle - Many client-specific interfaces are better than one general-purpose interface.
  • Okay, so this isn’t an interface. ISP is definitely meant for solving another problem. However, the basics behind this principle apply here. This is a general-purpose class. Why should I have my code strongly-coupled to this class if all I need is one method within it?

Now, to tackle some of the other arguments in favor of helper classes, let’s look at another example I’ve seen in the past:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public class DatabaseHelper
{
    private static DataAccessObject daoInstance;

    public static DataAccessObject CreateDataAccessObject(User invokingUser)
    {
        if (daoInstance == null)
        {
            var dao = new DataAccessObject();
            dao.AuthenticationMode = AuthenticationModes.Shared;
            dao.Credentials = new DataAccessCredentials(invokingUser.Kerberos, invokingUser.Token);
            return dao;
        }
        return daoInstance;
    }
}

Ah, rather than a helper, you meant a factory. Okay, I can dig that…sorta. We’re still using static here, which makes it impossible for me to factorize the creation of my DAO without strongly coupling my business logic to this specific class. Now, a factory is meant to abstract away the details of creating an object of a specific type. There’s some generally common expected inputs, and you get a generic implementation at the output. But, how should you go about doing that?

If you’re using a proper dependency injection/Inversion of Control container, you’ll be able to configure your application instance to inject a provider which implements the following interface:

1
2
3
4

public interface DataAccessFactory
{
    DataAccessObject CreateDataAccessObject(User invokingUser);
}

Furthermore, if you’re in a spot where you need a singleton instance of the DataAccessObject, then you can make sure that you implement the appropriate logic in your Factory instance and setup your IoC container to maintain a singleton instance of the object.

When discussing the use of “helper” classes, it’s more difficult to outright say they aren’t useful, because each person’s definition of helper classes are different. Nevertheless, if you venture down the path of bucket “utility” classes, stop. There are better approaches you can take.

Well, I still don’t agree with you…

That’s fine. You’re entitled to your opinion. Just don’t expect for me to pass your code reviews.

Are there circumstances where the use of a static method might be worth it? Quite possibly. But I rarely encounter those cases. If one of those reasons is “it’s faster to do it with a static than via good architecture” then you’re just being lazy. Yes, software development is about delivering value. However, what debt you introduce to the system must later be addressed save you be unable to deliver any value once you’ve built a castle with no doors around you.

I’m not alone in this. Check out just a few other blogs which carry several really good arguments against the use of statics or helpers:

• Anti-patterns and Worst Practices - Utils Classes
• Util Classes Must Die
• Are Helper Classes Evil?
• Helper Classes are Evil

Where did you shape your thoughts on best practices?

In my previous role at Improving Enterprises, there were a number of really talented technologists, passionate about best practices. They were staunch supporters of user groups and community as well. Thus, my opinions on best practices have been shaped not only through previous co-workers and colleagues, but also by community.

Community has a lot of benefits, and has a lot of diversity of thought. The software community provides an atmosphere for technologists to actively try many of the best practices they preach through katas, discussions, pull requests, and many other channels. These best practices are most commonly focused around the use of patterns. Why reinvent the wheel when patterns exist to help solve the same or similar problems? However, these patterns often have justifiable use: they leverage the facets of object oriented languages in the manner they were expected to be used.

This is where I started to learn about best practices. My source of community was extremely diverse. It included my colleagues at work, my close developer friends, and those technologists in the community who were extremely passionate about their work. Some of these technologists I engaged only through user groups, some through their blogs, and some through discussion boards. I am a high achiever, and in order for me to achieve, I must learn - so I am always on a continuous search for new information - even if it contradicts previous thoughts or practices I held imporant to me.

Finally, the only other way to really learn and solidify the importance of a best practice is to, well, practice it. With enough practice, you’ll invariably run into other practices which box you into a corner or make your life very difficult. The more times you encounter these situations, the greater a believer in best practices which leverage the power of object oriented design and development.

Diversity of thought is a great practice and should be encouraged. It leads to more out-of-the-box thinking, approaches to problems, and ensures that all risks are vocalized. However, one of the most important things I’ve had to learn is that no matter what, diversity of thought should always be supported with sound reason and justification. Simply citing a preference to a mode of practice, idea, or solution without good justification to a practice is hard to accept in a professional setting.

[2] http://www.improvingenterprises.com/ [3] http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)

I’m a huge fan of the e-mail brevity challenge. The idea is simple: keep your e-mails short. 140 characters short. What this limits you to do is huge. Do you think you can have technical design discussions with 140 characters? What about researching technical support problems? How about product feature ideas? Clarifying how your team came to the conclusion to disable a feature due to security constraints? You might be able to do this in a bunch of 140 character e-mails, but that’s hardly a good use of time.

The Problem

For those examples I listed above, how important is it to your team to find that information now or three years from now? Let’s face it - most teams undergo turnover at some point and with some degree of variability. Those whom are left find themselves struggling to find the context - the why - beind a particular decision. This becomes more evident the longer time passes since a decision was made.

We have this problem now. I am considering simplifying a system we have currently. Yet, I can’t find any context to understand why the system is currently so complex. Everyone agrees there should be a reason, but nobody knows why that reason is. The how of the system was perfectly documented, but the why was not. If current (and former) history are any sign, it’s likely the conversation of why has been lost somewhere along the way.

The Solution

As previously indicated, expressing yourself via e-mail in 140 characters or less is hard. You want to know what isn’t hard to express under that constraint? A URL.

The solution to this problem is easy. If you shift your conversation online, its longevity or persistance is no longer tied to the life of someone’s inbox. We have what we call a “posting” culture at the firm - a process to keep as many people who are directly involved or may be involved informed about the progress of our work. However, doing so via e-mail is laborious, especially if you leave out a stakeholder. However, if the conversation is non-private, having that conversation on an online social platform enables those stakeholders to opt-in to your updates. They can sit on the sidelines watching the activity, or if they see something that requires their intervention, they can take charge and engage in the conversation.

But what about private conversations? Most social platforms have the capability for engaging in private conversation. Whether this be through closed collaboration/team rooms or direct messages between you and another user. However, once decisions have been made, it becomes easy to promote the relevant pieces of those conversations from private to public - it’s already online.

Additional Benefits

The problems of traditional e-mail go beyond simply lack of context. Many corporate e-mail platforms still don’t provide metadata tagging. Searching for e-mail means you search conversations only you were directly involved. Being able to keep track of the conversation thread is dismal with our tools - only a few e-mail clients (and only very recently) are able to present conversations in threaded views. Most people still don’t know those features exist! Finally, collboration on documents still happens by shipping them via e-mail. Which is the authoritative final version? Finally, what was the final decision of that chain of e-mails about how to proceed on our design?

Moving the conversation online to social platforms opens up many other benefits. Tagging (primarily #tags, pronounced as hashtags) are in most platforms. For those not familiar, tagging content with certain keywords enables easier discoverability. When you search for relevant conversations using those keywords, the search engine considers tagged posts more relevant than posts having those words somewhere in the document. A human made the decision that keyword pertains to this conversation, rather than that word being used in a conversation about something completely different. Search is also enhanced as you’re now searching all open and public conversations, broadening your search potential.

Being able to follow conversations is much easier. Threaded views enable clear visibility and understanding of the flow of conversations. Furthermore, most social platforms (especially in the Q&A space) have options for marking something as a “correct” or “most helpful” answer. Leveraging these same features, even for team discussions accomplishes two things:

• You know exactly which decision was made, even when the content indicating that a decision had been made was unclear.
• You have all of the conversation surrounding why that decision was made.

Finally, when it comes to sharing and collaborating on content, like Word/Excel/PowerPoint documents, moving that online and providing a single authoritative source for the document reduces confusion on which is the final version and the final copy.

Give it a Try

Don’t wait. Force yourself into this model. It will be hard - you’ll want to revert back to your old ways of typing lengthy e-mails and using distribution lists. When you find yourself writing even just two sentences in an e-mail: stop. Immediately move to your online space and continue. Don’t be tempted to reply in e-mail to someone sending you the link to the topic thread they started online - keep the conversation where it originated! But move it online if it started in e-mail!

While your team starts adopting this pattern of communication, you’ll see some short-term gains (like discoverability, transparency, clarity of decisions, inclusion, etc.). However, this is all about the long-term gains your team will have. The more content you push online, the better it will be for your team when it comes to retaining critical information. It will be worth it!

• My Site Instantiation Interactive Request Queue
• My Site Instantiation Non-Interactive Request Queue
• My Site Second Instantiation Interactive Request Queue

The problem is, those timer jobs were missing from my server!

I checked for the timer jobs first through Central Administration, and then subsequently thru PowerShell. The following are the results I found:

1

Get-SPTimerJob | sort Name | ft Name

I decided to check for the My Site timer jobs on a know working SharePoint 2013 farm. Sure enough, they were there:

I wanted to look for a way to re-register timer jobs. There should be no reason for me to do something drastic like uninstall/reinstall the User Profile Service just to get the timer job re-registered. This timer job, as expected, was in the Microsoft.Office.Server.UserProfiles assembly:

So, I cracked open dotPeek and decided to hunt for that timer job definition and find who was using it. It turns out, there’s a web application-level feature (MySiteInstantiationQueuesFeatureReceiver) which registers these timer jobs:

There is a public static method inside that feature receiver used for registering the timer jobs.

So, I had one of two options: invoke the static method directly, or simply toggle the feature. I opted to toggle the feature:

1
2
3

$feat = Get-SPFeature 65B53AAF-4754-46D7-BB5B-7ED4CF5564E1
Disable-SPFeature $feat -Url http://webapp
Enable-SPFeature $feat -Url http://webapp

Finally, I double checked and verified the timer jobs were now present!

I then went to the My Site host URL and got into the queue for provisioning. Within 5 minutes, my personal my site was provisioned!

1

The target of the operation was not found. Internal type name: Microsoft.Office.Server.UserProfiles.SocialDataStoreException. Internal error code: 0.

The problem seemed fairly straight-forward. The targeted site I wanted to follow couldn’t be found. However, it didn’t make sense. The instance URL I was targeting (https://chrisweldon.sharepoint.com) did in fact exist. Furthermore, I was not already following that site. So, why was this failing?

Let’s take a look at some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

SP.SOD.executeOrDelayUntilScriptLoaded(userProfilesLoaded, 'SP.UserProfiles.js');

function userProfilesLoaded() {
    SP.SOD.executeFunc('userprofile', 'SP.Social.SocialFollowingManager', followSites);
}

function followSites() {
    var context = SP.ClientContext.get_current();
    var socialManager = new SP.Social.SocialFollowingManager(context);
    var socialSite = new SP.Social.SocialActorInfo();
    socialSite.set_contentUri("https://chrisweldon.sharepoint.com");
    socialSite.set_actorType(SP.Social.SocialActorType.site);
    socialManager.follow(socialSite);

    context.executeQueryAsync(
        function() { alert('Sites followed!'); },
        function(sender, args) { alert('Error: ' + args.get_message()); });
}

This is fairly straight-forward. I simply setup a social actor to follow and call out to the SocialFollowingManager to attempt to follow that site. This is when I thought, perhaps this is a permissions problem? I’m trying to have the JavaScript object model write a request to the Social datastore. Perhaps it wasn’t authorized.

I changed the permissions in my AppManifest.xml from Read to Write:

Sadly, I received the same error message. However, if you read the MSDN Documentation on Developing Social Features in SharePoint 2013, there is a section in there talking about user profiles:

User Profiles (http://sharepoint/social/tenant) The permission request scope used to access all user profiles. Only the profile picture can be changed; all other user profile properties are read-only for apps for SharePoint. Apps that request rights for the User Profiles scope must be installed by a tenant administrator.

In a nutshell, I have to grant Tenant permissions to my app to be able to have my user follow a new site. Therefore, my new permissions look like the following:

As indicated by the paragraph above, the app now needs to be installed by a tenant administrator. However, in doing so, the app now follows sites (and other content) with ease. Why the obscure error message? That I don’t know, and I hope the SharePoint team might look to address this with a more correct error message in the near future.

The recommended path to upgrade from 2010 to 2013 is a content and service-application database migration. This works great for us since we have to do this piecemeal. However, many of the general approaches for converting to claims-based authentication is to do so at the web-application level, rather than the content-database level (source: TechNet).

In SharePoint 2013, there’s actually an SPWebApplication method dubbed MigrateUsersToClaims that takes 3 arguments:

• NTAccount
• removePermissionsAfter
• SPWebApplication.SPMigrateUserParameters

There was no guidance on the NTAccount, other than the user “performing” the operation. I opted to use the farm account to ensure it had the appropriate level of permissions. The true power of the content database migration comes in with the third parameter. We can add individual content databases to migrate with this parameter rather than worrying about the entire web application.

Props go to Steve Peschka who originally pointed this out. However, in his post, the PowerShell to do this upgrade was the following:

1
2

-- SNIP -- 
$wa.MigrateUsersToClaims($acc, $true, $arguments)

For me, that throws the error Exception calling "MigrateUsersToClaims" with "3" argument(s): "Operation is not valid due to the current state of the object." This was strange, and I couldn’t figure it out. So, I cracked open the Microsoft.SharePoint.Administration dll and took a look at the MigrateUsersToClaims method:

1
2
3
4
5
6
7
8

public bool MigrateUsersToClaims(NTAccount account, bool removePermissionsAfter, SPWebApplication.SPMigrateUserParameters parameters)
{
    if ((NTAccount) null == account)
        throw new ArgumentNullException("account");
    if (removePermissionsAfter && parameters != null && (parameters.HasDatabaseToMigrate || parameters.OnlyMigratePolicyUsers))
        throw new InvalidOperationException();
    // The rest
}

That second one was the one that I questioned. I know the conditions matched for the first two checks. The question was how my parameters looked. Sure enough:

1
2
3
4
5

PS > $arguments

DatabasesToMigrate      HasDatabaseToMigrate        OnlyMigratePolicyUsers
------------------      --------------------        ----------------------
{WSS_MigrationTest_...}                 True                         False

With that, if I changed the middle parameter from $true to $false, the migration finally ran (and completed) succesfully.

Why did this happen? This was because my $acc user is my farm account. I’m also running my PowerShell session as my farm account. This is to ensure that I have full, unfettered access to the SharePoint Object Model and the Content Database. The middle parameter states (from MSDN):

The **account** will be given the correct permissions to perform the operation. Should this permission be removed when the operation is complete.

We definitely don’t want this for the farm account. So, my updated code, for reference:

1
2
3
4
5
6
7

$wa = Get-SPWebApplication http://my-app-url
$acc = "domain\farm-account"
$arguments = New-Object Microsoft.SharePoint.Administration.SPWebApplication+SPMigrateUserParameters

$contentDb = $wa.ContentDatabase["Content Database Name"]
$arguments.AddDatabaseToMigrate($contentDb)
$wa.MigrateUsersToClaims($acc, $false, $arguments)

Cheers. Once again, thanks go to Steve Peschka for figuring this out.

The application required Tenant Read permissions as I’m accessing the User Profile through the JavaScript object model. The purpose of using the User Profile JSOM is to get the suggestions of sites and people to follow that SharePoint presents.

Now, the concept of a “Tenant” makes sense for Office 365 or SharePoint Online. As a hosting provider, there are multiple tenants you want to support in a single environment. However, for an on-premise deployment, this error message didn’t make much sense. I started poking around and came across spinning up a tenant administration site, being able to set multiple app tenants through the App Management Service cmdlets, but none of those really seemed like the right solution for my on-premise deployment. I found an MDSN Forum Question which seemed closer to the solution. That post recommends splitting the service accounts used to host the App Management and Site Settings services from the farm account. This was critical as the Farm Account is not allowed to add apps under its identity whatsoever. You will get an error message when trying to provision, and the ULS logs will indicate that an assertion failed checking that the current account was not the system account.

What did it turn out to be? I just needed to make sure my user account was directly added as a member of the Farm Administrators group. We have traditionally deployed farm administrators via an Active Directory and local (Administrators) group. However, it appears that the App Management service dislikes this approach and wants users explicitly permissioned to the Farm Administrators group. Additionally, after granting your user direct permissions, you need to issue an iisreset so those changes take effect. Then, you can provision your app successfully.

When moving this workflow into my customer’s environment, they require I specify a JIRA ticket on each commit to the repo. I had forgot to do that for the first few commits, which looked like this:

1
2
3
4
5
6

$ git log --oneline

6fa0719 Fixing the handlebars template to render the target as parent.
21ad0cf Initial commit of the SiteSuggestions app
6430b3b Adding the .gitignore files
095d92e SP-34: Creating branch to track the new suggestions app part.

So, when I tried to commit, I got an error indicating that the commit failed. In looking at the error, I saw the pre-commit hook error requiring that I associate the commit with a JIRA ticket. So, my first comment was easy to fix:

1
2
3

$ git commit --amend -m "SP-34: Fixing the handlebars template to render the target as parent."
[2013-sitesuggestions-svn 6b4bc64] SP-34: Fixing the handlebars template to render the target as parent."
 1 file changed, 1 insertion(+), 1 deletion(-)

Now I just had to figure out how to modify commits 21ad0cf and 6430b3b. I tried to use git interactive rebase as per the git svn tutorial and per the git-scm book. However, in both cases, I got the following error message:

1
2
3

$ git rebase -i HEAD-3
fatal: Needed a single revision
invalid branch HEAD-3

The solution was to specify the SHA for the commit:

1

$ git rebase -i 6430b3b^

This started the rebase process and allowed me to amend the commits for every faulted commit. Now, everything looks correct:

1
2
3
4
5
6

$ git log --oneline

6fa0719 SP-34: Fixing the handlebars template to render the target as parent.
21ad0cf SP-34: Initial commit of the SiteSuggestions app
6430b3b SP-34: Adding the .gitignore files
095d92e SP-34: Creating branch to track the new suggestions app part.

When I finally committed using git svn dcommit, everything went perfectly!

I’m used to having the ribbon to edit pages in SharePoint, regardless if they are standard publishing pages or web part pages. However, in the “My Site Host”, there is no ribbon for the standard pages, even when you are a farm administrator. I jumped to the conclusion that I could add the App Part via SharePoint Designer. Sadly, this wasn’t the case. SharePoint Designer does not list any app parts.

I tried to go through the rigarmarole of adding the app to a separate site through the web editor, then copying the code from within SharePoint Designer and pasting it into the Newsfeed page, only for that to fail. The identifiers for the apps are completely different.

This is when I stepped back and though that the solution should be simpler than this. It turns out, it was. Click the gear icon (settings menu) in the upper-right corner of SharePoint and you’ll find the Edit Page link. I felt liberated and frustrated at myself for not checking there earlier. From there, you have complete control to edit the Newsfeed web part page (or any page in the “My Site Host”).

Authentication

All requests currently are performed using HTTP Basic authentication. While we are still developing our OAuth provider and intend to use that as our primary mechanism for authenticating users, we currently are accepting your username and password via a standard HTTP Basic Authorization request. As a raw HTTP header, you are looking at the following for authenticating as demo@quicktate.com with password Test1234%:

1

Authorization: Basic ZGVtb0BxdWlja3RhdGUuY29tOlRlc3QxMjM0JQ==

As a cUrl request, you’ll be looking at using the -u demo@quicktate.com:Test1234% argument to the command.

Transcription Requests

The basis of Quicktate is transcriptions. However, while there is a /transcription resource for our API, you don’t actually create transcriptions - that’s the job of our typists. Instead, you submit a /transcriptionrequest to Quicktate and operate on that request until a typist has completed transcribing and turns it into a /transcription.

So, we expect very little in the HTTP headers for this type of request. Because you’re creating a new Transcription request, the method will be POST. What’s most important is the body of the request. Most developers are used to submitting the request as a form request, to where in a PHP application, the variables will come across in $_POST. In this case, however, we expect the body of the message to be either XML or JSON (as indicated in the Content-Type header).

What should be in the body of the message? The transcription request. When creating a transcription request for the first time, you’ll need to supply us with the following information:

So, a sample HTTP request might look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

POST /v1/api/transcriptionrequest HTTP/1.1
Authorization: Basic ZGVtb0BxdWlja3RhdGUuY29tOlRlc3QxMjM0JQ==
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: api.quicktate.com
Accept: application/json
Content-Type: application/json
Content-Length: 208

{ 
  "callbackDestination": "https://private.host.callback.com/callback-url.php",
  "callbackMethod": "HTTPPOST",
  "metadata": "My custom metadata",
  "language": 1,
  "audiourl": "http://www.quicktate.com/audio.wav"
}

or from cUrl it would look like the following:

1

curl -u demo@quicktate.com:Test1234% -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{ "callbackDestination": "https://private.host.callback.com/callback-url.php", "callbackMethod": "HTTPPOST", "metadata": "My custom metadata", "language": 1, "audiourl": "http://www.quicktate.com/audio.wav" }' -v https://api.quicktate.com/v1/api/transcriptionrequest

The response I’m likely to receive back will look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

HTTP/1.1 201 Created
Server: nginx
Date: Mon, 28 Jan 2013 06:14:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.17-1~dotdeb.0
Vary: Accept
Access-Control-Max-Age: 86400
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, X-Authorization, Origin, Accept, Content-Type, X-Requested-With, X-HTTP-Method-Override
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, HEAD, OPTIONS
Location: /v1/api/transcriptionrequest/14967237

What’s important to realize is that you will only have a successful request submitted when the status code returned is a 201. If you get a 400-level error message, check which one. If it’s a 401 or a 403, then you are not sending the proper credentials. If it’s just a 400 error, then you need to look at how you’re sending your data to the API, because the API doesn’t recognize the payload.

Once you’ve gotten past that, you may be wondering, how do I access the transcription request now? There is no body to this whatsoever. This is one of those REST semantics at play. If you look in the return headers, there is a Location header which specifies exactly where you can access your transcriptionRequest. Simply submit a GET request to that URL and you’ll find the status of your transcriptionRequest. The response will look something similar to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

HTTP/1.1 200 Success
Server: nginx
Date: Mon, 28 Jan 2013 06:14:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.17-1~dotdeb.0
Vary: Accept
Access-Control-Max-Age: 86400
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, X-Authorization, Origin, Accept, Content-Type, X-Requested-With, X-HTTP-Method-Override
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, HEAD, OPTIONS

{
  "id": 14967237,
  "callbackDestination": "https://private.host.callback.com/callback-url.php",
  "callbackMethod": "",
  "status": 0,
  "metadata": "Represents an unprocessed audio file. ",
  "datePosted": "2012-12-01T01:23:45-0600",
  "language": 99,
  "audiodata": null,
  "audiourl": null
},

Completed Transcriptions

That is, until your transcription is complete. Once your transcription is complete, the resource URL will permanently move. As a result, you’ll see the status code for the previous URL change from 200 to 302, indicating that it’s been permanently moved. Fortunately, we point you in the direction of where you need to go again, through the Location header:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

HTTP/1.1 302 Moved
Server: nginx
Date: Mon, 28 Jan 2013 06:14:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.17-1~dotdeb.0
Vary: Accept
Access-Control-Max-Age: 86400
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, X-Authorization, Origin, Accept, Content-Type, X-Requested-With, X-HTTP-Method-Override
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, HEAD, OPTIONS
Location: /v1/api/transcription/14967237

When you submit a GET request to the Location URL listed above, your payload will look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

HTTP/1.1 200 Success
Server: nginx
Date: Mon, 28 Jan 2013 06:14:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.17-1~dotdeb.0
Vary: Accept
Access-Control-Max-Age: 86400
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, X-Authorization, Origin, Accept, Content-Type, X-Requested-With, X-HTTP-Method-Override
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, HEAD, OPTIONS

{
  "id": 14967243,
  "metadata": "Represents a processed audio file. ",
  "datePosted": "2012-12-01T01:23:45-0600",
  "dateCompleted": "2012-12-01T01:25:30-0600",
  "wordcount": null,
  "language": 99,
  "transcription": "This is a completed transcription request."
}

More to Come

Hopefully by next week, I’ll have another writeup on how to interact with your transcription requests before we begin processing them.

I double (and triple) checked that the permissions for the account executing the code were correct. Since we’re using SharePoint, we are using only Windows authentication. Furthermore, I made sure that our connection string was using Integrated Security:

1

Server=sqlserver01;Database=Journaling;Integrated Security=true;

Ultimately, it came down to the fact that the SQL Server was not setup to allow remote server connections to filestream. I managed to fix this by following the Enable and Configure FILESTREAM article on MSDN. The Allow remote clients to have streaming access to FILESTREAM data. box was not checked, thus causing my problems.

The problem was first noticed by an early adopter of the new RESTful API we released a month ago. He kept receiving an HTTP error code 400 with the following message:

1
2
3

{
    "message": "Invalid Payload Format"
}

When I dug into this, I found this error was coming from the RESTful Zend Framework component I was using to develop our RESTful web service. However, when I looked at the customer’s request payload, it looked fine (the audiodata parameter was about 60k in size, properly Base64 encoded):

1
2
3
4
5
6

{
  "callbackDestination": "chris@chrisweldon",
  "callbackMethod": "EMAIL",
  "language": 1,
  "audiodata": "gEUS9gckvq37RG4J7LYg12YdZAwVJCB053EiDJp9iNLDr6vGyOWvFDzyPwqVoS6UVO+ ... etc"
}

However, when I started digging in and debugging this issue, I saw that about half the web requests being sent to my php-fpm backend were malformed. The payload looked like this:

1
2
3
4
5
6
7
8
9
10

{
  "callbackDestination": "chris@chrisweldon",
  "callbackMethod": "EMAIL",
  "language": 1,
  "audiodata": "gEUS9gckvq37RG4J7LYg12YdZAwVJCB053EiDJp9iNLDr6vGyOWvFDzyPwqVoS6UVO+ STOPPED SHORT"
{
  "callbackDestination": "chris@chrisweldon",
  "callbackMethod": "EMAIL",
  "language": 1,
  "audiodata": "gEUS9gckvq37RG4J7LYg12YdZAwVJCB053EiDJp9iNLDr6vGyOWvFDzyPwqVoS6UVO+ EVEN SHORTER

This was a very strange pecularity. The body of the message was being duplicated, but the audiodata parameter was being cut short in two different places in the request. I presumed this was a problem with nginx, but I couldn’t find any information to guarantee this. Furthermore, the behavior seemed rather erratic. Some requests would include the full payload, others would have this behavior from above.

That’s when I took a look at my syslog. I started to see the following errors in /var/log/syslog:

1

Jan  6 16:24:53 app01 kernel: [4435039.857398] php5-fpm[19547]: segfault at 7fff7b2a8fe0 ip 000000000048f7dc sp 00007fff7b2a8fd0 error 6 in php5-fpm[400000+824000]

I checked the /var/log/php-fpm.log and saw a similar output there as well:

1

[06-Jan-2013 16:24:53] WARNING: [pool www] child 19547 exited on signal 11 (SIGSEGV) after 450.542383 seconds from start

Segmentation faults? This seemed rather extreme just for handling 60k of payload.

I continued digging further and figured out what was going on. I managed to find that the first request passed to the application servers contained the complete payload, properly Base64 encoded. However, this request was what triggered the segmentation fault. Once that segmentation fault was triggered, the signal abruptly terminated and nginx re-dispatched the request to a separate application node. This application node was receiving the malformed body, not causing a segmentation fault, and was instead returning the Invalid Payload Format message.

So, to get to the bottom of what was going on, I started throwing logging statements into my application to see where I got to. Eventually, I ended up at a validation block of code, to validate that exact parameter that was having the truncation problems (audiodata):

1
2
3
4
5
6
7

if (trim($value->audiodata) != '') {
    // Verify the data is base64 encoded.
    if (!preg_match("/^(?:[a-z0-9+\/]{4})*(?:[a-z0-9+\/]{2}==|[a-z0-9+\/]{3}=)?$/i", $value->audiodata)) {
        $this->setMessage('The audio data supplied was either not Base64 encoded or was incomplete.');
        return false;
    }
}

It turns out, preg_match has a configurable maximum upper limit on the number of characters (source). This really makes sense, as regular expression matching should be on small bodies of text (generally, < 10k characters is my good recommendation). Therefore, this was not a suitable solution. We would be better off attempting to base64 decode the text, and if it fails, assume it’s invalid. This can easily be done with the following:

1
2
3
4
5
6
7

if (trim($value->audiodata) != '') {
    // Verify the data is base64 encoded.
    if (base64_decode(chunk_split($value->audiodata)) == false) {
        $this->setMessage('The audio data supplied was either not Base64 encoded or was incomplete.');
        return false;
    }
}

After deploying this code change, all is well. No more segmentation faults.

The last quarter I eased back on my speaking engagements, largely because I needed to spend more time with my family and because conferences are fairly light in the third quarter. Nevertheless, I woke up on January 1, 2013 to a rather pleasant e-mail - I had been awarded a Microsoft MVP in ASP.Net/IIS. To say this came as a surprise would be a lie, but I had been skeptical if I was going to get the award, given how competitive the awards are. There tens of thousands of professionals all over the world vying for one of these awards, and in some categories (C# and SharePoint, in particular), you have to go through multiple vetting cycles before you end up receiving the award.

For those unfamiliar, an MVP Award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others.. No doubt, that’s how I see myself (though a little more modest). I realize that there are some people who only care to get awards such as these as yet another way to pad the resume. Those individuals most definitely do what they do for the wrong reasons.

I do what I do for a number of reasons. First, I love to hear myself talk. Just kidding. Those who know me know I can be (somewhat) shy in the company of others, and never as boisterous as some of those in a clique (I’m looking at you Devlin and Bud :-P). That said, I do love meeting new people. I’ve forged some really great relationships with people in the community that I would love to work with (and some of them now do!). Even if I never have that opportunity, being able to meet for beers after work with these people is a whole lot of fun.

But, while meeting new people is fun, I consider speaking as a selfless act. I don’t speak for myself - I speak for you, the community. If nobody wanted to hear what I had to say, then my conference rooms would be empty and I’d likely stay behind my computer or doing other things to help out. However, the fact that I’ve concentrated on putting together such high-quality, passionate talks on a variety of different technical and professional areas has proven to be useful to many people in my audience. What matters most is having an impact on at least one person’s perception of a topic, whether that’s opening their eyes to a new technology, providing a different perspective on how to write software, or simply inspiring them to try again at something they previously failed at - that’s what I really love.

How I View the Award

Some people see the award as an affirmation that they are the most elite, technically proficient individiuals in the world. I won’t lie - that’s how I’ve always percieved Microsoft MVPs. Some of them are stellar individuals who really are at the top of their game. However, to assume that I’ve reached the pinnacle of my technical abilities is far from the truth. I thought I was a hot shot coming into Improving Enterprises; working with such a talented, professional, and wonderful group of technologists and mentors has really helped me remain consistently modest about my technical abilities.

In our field, there are experts, people who cannot achieve a level higher than they are already at. However, those individuals are experts on such a niche topic, that field itself is already evolved to its maximum potential, allowing for experts to finally attain their place. However, I see myself as extremely proficient in a lot of the areas I work - SharePoint, PHP, ASP.Net, and systems administration, in particular. That said, because each of those areas are so broad and have so many different uses, it’s impossible for me to become an “expert” in these fields. It’s hard for anyone to become an expert in these fields. If you come across anyone who claims to be an expert, I guarantee you’ll find gaps in their knowledge somewhere. That’s how I view experts - people who know everything there is to know.

With that said, I am humbled to know that Microsoft does see me as not only a skilled technologist, but that my contributions to the community are useful. That’s ultimately how I view this MVP Award - it’s affirmation that I should continue to do what I’m doing. It’s Microsoft willing to expend resources and provide me with information that will continue to help me educate my audience.

I don’t (and won’t) hold my MVP award above anyone else - that’s just not the type of person I am. Instead, I proudly wear this badge of honor as a beacon for others to reach out to ask questions. I’m always interested in helping people out - that’s why I work for Improving Enterprises. Their motto directly aligns with my goals - Improving - It’s what we do.

Thanks

I would not be receiving this award if it weren’t for the help of Improving Enterprises. The company actively encourages community participation, and goes above and beyond to help each other out with mentoring, guidance, and access to some of the best people in our area. There are two people from Improving who have been great motivators and put their necks on the line to vouch for me - Devlin Liles and Tim Rayburn. You guys rock and I love working with you both! Allen Hurst, my mentor for the last two and a half years, has truly been an inspiration, and has become one of my best friends. Without him, I likely wouldn’t have been such an avid speaker. Zain Naboulsi and Chris Koenig, both from Microsoft, were also a great help towards getting me vetted through the MVP process and deserve major kudos.

Of course, I would not be nearly as successful at the things I do without my family. I thank my father, David Weldon, for showing me the way early in being a speaker. For as long as I can remember, my father not only attended medical conferences, but was frequently a speaker at them, and often times went out of his way to make sure that I could attend to watch him speak. While I rarely ever understood the material, his charisma and style of presenting have rubbed off - in a very positive way. Thank you, dad.

Finally, to my wife, Melissa, and son, Tristan, you both deserve the sun and the moon. You both motivate me to continually be a better person, encourage me when I am under stress, and are understanding when I am not around. I am often times not around or am too busy to spend the quality time you two deserve, and for that I’m sorry. I shout from my proverbial mountain top how much I love you both, and how much you both mean to me, and thank you both so incredibly much for being such a wonderful and patient family.

Update (25/Apr/2013): Microsoft has implemented a bugfix for this. See below for more information.

Background

This blogpost (and the ordeal mentioned above) all started with the community discussion thread seen in the screenshot above. In order for all social features to work correctly, you have to have the following services configured correctly in your SharePoint 2013 environment:

• Managed Metadata Service
• Search Service
• User Profile Service

Each of these are used in a variety of different ways to knit the social story together. Let’s start with the lowest common denominator in the tagging story: tags. If you read my previous blog post, you’ll get a better understanding of what Tags & Notes are used for in SharePoint. Through that interface, you can tag a list item, file, or even a list with any managed metadata that’s marked “taggable”. The purpose is to help surface relevant content within an enterprise without going through the hassle of creating custom metadata columns on document libraries.

In SharePoint 2013, there is a new managed metadata term set, dubbed Hashtags. This is a system-level term set, and cannot be deleted or created. However, all terms created within are fully manageable. This term store contains the terms created when a user creates a tag beginning with a hash (#). This occurs no matter where the user tags something: a file, comment, microblog post, or a community discussion. Suppose for a moment that a user adds a community discussion post with four new hashtags: #reply, #outlook, #metadata, and #termstore. Upon submitting that post, those terms will immediately be added to the ECMTermLabel database table in the Managed Metadata Service database.

1
2

SELECT TOP 1000 [PartitionId], [TermId], [LCID], [Label], [IsDefault]
FROM [dbo].[ECMTermLabel]

However, all across SharePoint and especially the search service, the terms are not refered to by their integer identifiers. Instead, they’re referred by their UniqueId guid. This is found by looking at the ECMTerm table, alongside the ECMTermLabel table:

1
2
3

SELECT TOP 1000 [Id], [UniqueId]
FROM [dbo].[ECMTerm]
WHERE Id IN (1088, 1089, 1090, 1091)

At this point, it’s important to rehash that hashtags are not social by default. When it comes to using the Tags & Notes board, it is possible to enable any tags used there (including hashtags) to be “socialized”. However, the “new” social features (community discussions, microblog posts and comments) do not make those tags social. You can check by querying the User Profile Service SocialDB’s SocialTags table.

Deeper Dive - Where are my Hashtags?

So, I wanted to find out where my hashtags really were being used. So, this took me on a deep dive through the SharePoint content database. As this was all read only, this didn’t really matter if I was poking around trying to find data. Because there are several tables with hundreds of columns, I wanted to hone in precisely the list I was looking for my hashtags: the Discussions List. I cracked open the AllLists table and queried based on title:

1
2
3

SELECT TOP 1000 [tp_SiteId], [tp_WebId], [tp_ID], [tp_Title]
FROM [dbo].[AllLists]
WHERE [tp_Title] = 'Discussions List'

The tp_ID is what I what I needed, as it is the list’s unique identifier for me to pull content out of the AllUserData table. This table contains all user-provided data in all SharePoint sites stored within this content database. Therefore, it’s massive, even for an environment with only a single site collection like a community site. Performing the following query allowed me to find the discussion comment that I added in the topmost screenshot.

1
2
3

SELECT TOP 1000 [tp_ColumnSet]
FROM [dbo].[AllUserData]
WHERE tp_ListId = '75D9F5D7-8B83-4252-BB52-48D5E5F91B01' ORDER BY tp_Modified DESC

This yields a field which is viewable in an XML editor. The resulting XML for the screenshot of the comment made by me in the post above looks like the following. As you can see in the ntext9 XML field, we have the hashtags that SharePoint has identified within the body of the comment, as well as their unique identifiers from the managed metadata service. Further proof that community sites are being tagged appropriately.

1
2
3
4
5
6
7

<int1>42</int1>
<int2>1</int2>
<int3>42</int3>
<int7>1</int7>
<ntext2>&lt;div class="ExternalClass27572E39547C412CA62D7D22B7FC380F"&gt;&lt;p&gt;Let's test. &lt;span class="ms-rtestate-read ms-socialentity" data-hashtag="00000000-0000-0000-0000-000000000000" data-hashname="#reply"&gt;&lt;a class="ms-hashTag" href="/_layouts/15/FeedRedirector.aspx?type=tag&amp;amp;value=%23reply" rel="#reply"&gt;#reply&lt;/a&gt;&lt;/span&gt; &lt;span class="ms-rtestate-read ms-socialentity" data-hashtag="00000000-0000-0000-0000-000000000000" data-hashname="#outlook"&gt;&lt;a class="ms-hashTag" href="/_layouts/15/FeedRedirector.aspx?type=tag&amp;amp;value=%23outlook" rel="#outlook"&gt;#outlook&lt;/a&gt;&lt;/span&gt; &lt;span class="ms-rtestate-read ms-socialentity" data-hashtag="00000000-0000-0000-0000-000000000000" data-hashname="#metadata"&gt;&lt;a class="ms-hashTag" href="/_layouts/15/FeedRedirector.aspx?type=tag&amp;amp;value=%23metadata" rel="#metadata"&gt;#metadata&lt;/a&gt;&lt;/span&gt; &lt;span class="ms-rtestate-read ms-socialentity" data-hashtag="00000000-0000-0000-0000-000000000000" data-hashname="#termstore"&gt;&lt;a class="ms-hashTag" href="/_layouts/15/FeedRedirector.aspx?type=tag&amp;amp;value=%23termstore" rel="#termstore"&gt;#termstore&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;&lt;/div&gt;</ntext2>
<ntext9>#metadata|9434622a-60e1-47c6-9b58-17b955c31410;#outlook|497ba856-65ff-45d7-89f6-6302e041e4b8;#reply|5c68c7c8-38cc-4964-850f-8c1e02076c04;#termstore|b2969adf-a746-4c0e-b38f-db7a77de7788</ntext9>
<nvarchar3>&lt;8b0edc16e721456986d9d04eac185d5d@SharePoint&gt;</nvarchar3>

The Bug

The root cause of me digging down this rabbit hole was no matter what I did, I couldn’t find content tagged with certain hashtags that I know existed. I had a tendency to use the #sharepoint hashtag a lot in both my microblog feed as well as in my community discussion. Yet, when I attempted to search for that hashtag through the search center user interface, only the microblog posts would show - none of the community discussion threads.

Furthermore, when I would check the hashtag profile for #sharepoint, it would show the same - only microblog posts, no community discussions. This is a significant usability issue.

I verified when creating a reply to a community discussion, complete with a new hashtag, the hashtag does get added to the managed metadata term set, as seen in the query from the Managed Metadata Service database above. In the ECMTermLabel database table, we can clearly identify the hashtag terms #reply, #outlook, #metadata, and #termstore added as part of the comment. As seen in the XML block from AllUserData above, we see that SharePoint is correctly associating the hashtags in the post with their identity in the managed metadata service. Just to make sure that the latest hashtags are exhibiting this problem, I performed a search and found nothing.

The biggest thing that I wanted to find out was whether Discussion Lists were properly surfacing hashtags as the appropriate metadata for the search service to identify them. When I looked at the list settings for the Discussions List, I found that it was affiliated with two content types: Discussion and Message. For the discussion list, we can clearly see that it is not directly using hashtags via the columns composition:

Furthermore, if you look at the Discussion and Message content types, neither reference Hashtags as a column:

Just to double check, I decided to look at the Microfeed for my personal site to see if the hashtags were being elevated as their own column. You can access your microfeed list by going to /personal/USERNAME/Lists/PublishedFeed. Indeed, when I looked at the columns for this list, Hashtags from the Managed Metadata Columns collection were being surfaced:

Summary

If this is indeed a bug, I can hope that Microsoft will fix this problem ASAP, as it’s a significant hinderance to groups adopting SharePoint 2013 for it’s social aspects. Despite all the marketing statements indicating that 2013 makes things much easier to find, this seems to be a limitation of SharePoint Social Search. I plan on creating a bug with Microsoft’s bugreporting system and will keep this post updated with information as it evolves.

Update (25/Apr/2013): I have received word from Jennifer Bester (@jbester), a Microsoft Field Engineer, that Microsoft has acknowledged this as a bug and has implemented a fix. It is due to be released in the June 2013 CU for SharePoint. As soon as that CU is published I’ll be applying it to our on-premise environment. SharePoint Online customers should be seeing this bug fixed ahead of the CU.

Improving is not nearly as large a company as the customer I work for. They are a financial institution with 40k+ employees worldwide. As a result, they have a distinct need to be social. But, they are also encumbered with some regulatory restrictions affecting their social deployment. As a result, introducing social across their firm is quite a challenge, especially when there are many stakeholders who are considering multiple competing products, including Jive, The Open Source Q&A System (OSQA), and Yammer.

A large part of social in the listed products and many other social networks is the concept of tagging. SharePoint 2010 started venturing down the path of tags by creating the Tags and Notes board, giving users the ability to tag files and lists publicly (or privately), making it easier to categorize items in SharePoint. You could also enhance your user profile by indicating what topics (Keywords) you should be asked about, giving users a sense that you’re the expert in those key fields. All in all, a good step in the social direction, but still behind the curve.

SharePoint 2013 continued to build on tagging with the introduction of #tags (prononunced hashtags). This is more beneficial to the social story for many reasons. First: hashtags become a first-class citizen in the sense that users no longer have to go out of their way to add a tag to something (like a conversation, comment, or microblog entry). Second: hashtags are always public, meaning that users have the collection of hashtags at their fingertips, giving them helpful hints on how to categorize their entry. Third: hashtags seem to be used anywhere you can tag - whether it’s in conversations, comments, or when tagging documents or libraries.

However, as I continued to dig into the social and tagging story, there seems to be some discrepancies in several of the features. The purpose of this post is to highlight that while #tags seem to be the direction that Microsoft would rather push users to use in their tagging strategy, integrating their existing social tagging strategy seemed to be left behind in SharePoint 2013.

Show Me More Tags

Tags & Notes

As mentioned above, there are several different ways to tag content in SharePoint 2013. The first is the original tagging strategy through the Tags & Notes board.

In this board, when you start typing text, you’ll get helpful hints of the various Keywords already available by means of other user contributions and pre-populated keywords from administrators of the Managed Metadata. What’s even more impressive is you can not only leverage Keywords from traditional tags, but also #tags:

Newsfeed

As everyone has been talking about, the Newsfeed is probably the singular most important feature to the social story. It’s the user’s “dashboard” into what’s relevant to them, what’s happening for everyone, and how to get to most other contextually-relevant content in the farm. If you read most of the marketing talk on the SharePoint 2013 social features, one of the biggest considerations was how to surface content that’s not just useful, but contextually relevant. Finding a sales report from the previous quarter may be useful in my position, but when I’m trying to search for content on the proposed social media marketing strategy, it isn’t useful.

From the newsfeed, I can quickly share status updates with everyone, my team, or other audiences of the sites I’m following. I can mention other users by using an @ symbol in front of their name and also have multiple #tags in a single post. At a glance, I also see information about the sites, documents, people, and tags that I’m following, making it easy to get around to find items that are most relevant to me. Searching for content is ubiquitous - you can pretty much search for content from anywhere in your SharePoint farm using the out-of-the-box templates.

Communities

Communities are the next biggest value-add to social adoption in SharePoint 2013. Communities are reminicent to forums and Q&A boards (like StackOverflow). Users can start threads in different categories of conversation and have rich discussion. #tags are ubiquitous, and can be added to questions and replies. I won’t go into the number of different features of communities, but suffice to say that I can easily add #tags to any of my interactions and they surface within searches and the newsfeed just like hashtags used in my microblogs.

However, one thing that appears to be less straight-forward is using managed metadata/keywords to tag individual responses. I can access the Tags & Notes button from the Page tab, but what’s less clear is what a “Page” is in the context of tagging content. Does it relate to the entire thread, or just the question? There are some unanswered questions around general social tagging for community discussions.

Off-topic: The biggest thing communities are meant to replace: distribution lists. Microsoft threatened to remove distribution lists from Exchange 2007 in favor of SharePoint, but backed off when they nearly had an enterprise revolt. They stayed away from the topic in Exchange 2010 and SharePoint 2010 (as far as I could tell), but 2013 seems to be a renewed effort. The message: communicating heavily through e-mail using Distribution Lists is a terrible approach. I’m in agreement. When I was a heavy Atlassian Confluence user, I found little point in communicating lengthy conversations in e-mail. Once I left, those e-mails would no longer be useful to future teams, and valuable business justification and decisions will be lost. Instead, I preferred to document and carry out these conversations in a Wiki (largly due to the E-mail Brevity Challenge of 2009).

Social Tags

What’s important to note, however, is the use of tags via the Tags & Notes board are not social tags by default. This is nothing new from SharePoint 2010, if you’ve already taken a look at this. What’s even more surprising is tags in communities or on your newsfeed are also, not social tags. To fix Tags & Notes, you have to go into the document library settings (specifically, the Enterprise Metadata and Keywords Settings section in order to enable the ability to save the metadata on this list as social tags. I have yet to figure out how to do this for communities or newsfeeds.

Personally, I feel that this is a missed opportunity. If Microsoft wants to solidify the social story more, then trying to surface the tagging of documents only helps that cause. Additionally, doesn’t it make sense that all #tags are social tags? If we’re using it in a social context, then it should be surfaced by the social controls. I digress, however, as the use cases for tagging vary widely and this may have been a strategic decision.

Why does marking a tag a social tag important? In SharePoint 2010 (and 2013), the only way you can surface a tag in your newsfeed is to follow it. To be able to follow it, the tag has to be a social tag. Not just any enterprise metadata can be followed in your newsfeed. For a large majority of metadata, however, this is totally fine: we likely don’t care to surface metadata from a term set dubbed SOX Compliance Restriction Codes showing ST123X-F.

Another reason social tags are important: the Tag Cloud. The tag cloud is a cool (but not innovative) feature to help surface the most popular tags in your subsite (or across the entire farm). The size of the text changes based on the number of uses of a particular tag. Nevertheless, other items like the tag cloud rely solely on social tags, not managed metadata tags. Therefore, when you drop one of those controls into a site, what’s rendered is based only on social tags. Crack open the code to check for yourself. I did.

Hashtags

This is the newest addition to the social tagging strategy in SharePoint 2013. #tags, like enterprise keywords and social tags, are a broadly defined concept meant to be leveraged in a wide capacity. Much like in social networking sites like Twitter and Facebook, the use of tags are to categorize a post, making it easier for individuals to search and identify related contents. All one has to do is add a hash in front of any word and SharePoint will convert that to a hashtag. When viewing the comment with a hashtag, SharePoint automatically renders it as a bold hyperlink to help make navigating and finding related content easier.

I regularly go to conferences that define a hashtag for people to use when posting to Twitter. This makes it easy for me as a conference organizer and an attendee to perform a search strictly on that hashtag. SharePoint is no different: they’ve realized that searching for hashtags and surfacing those results is strategically important to help further the social adoption in 2013. We see this through the hashtag profile pages (accessible by simply clicking the hashtag link) and through featured content controls in the search center.

Other controls exist (particularly in the newsfeed) that surface trending hashtags, which is useful to help bring visibility to important conversations or activities within the company.

Hashtags and Social Tags in the Managed Metadata Service

However, where I find the disconnect with the social strategy is that not all hashtags are social tags. Furthermore, not all Managed Metadata / Enterprise Keywords are social tags. Yet, most of this content all ends up in the Managed Metadata Service, but in two distinctly different System Term Sets: Hashtags and Keywords.

Even in our simple test environment with 7 enterprise users testing the social features, it became quite easy to cross-polute our term sets. While all of our terms in the Hashtag set are definitely hashtags, many of the terms in the Keywords set can be clasified as hashtags.

What’s really disappointing about this is the fact that none of those tags in either are guaranteed to make it to the social tags. Those are stored completely separately from the Managed Metadata Service. Social tags are an aspect of the user profile service and are stored in those databases. This is where things start to get very confusing. If you crack open the User Profile Service Social Database, there’s a SocialTags table that contains all tags used for Social elements. As you can see in the below results, these include regular keyword metadata tags as well as hashtags.

There are a couple of ways this is relevant. I can choose to follow a hashtag in SharePoint quite easily. In fact, SharePoint does an excellent job surfacing how to follow hashtags on your newsfeeds. While I can also follow social (not hash) tags, there is a distinct disconnect. If I tag a document with the SharePoint tag, if I’m following the #SharePoint hashtag, I won’t see that document in my newsfeed. I have to be following both the SharePoint tag and the hashtag.

This also affects search. As seen above, I can search for a hashtag and see all content tagged as a featured search result. However, searching for standard social tags can only be done through the tag profile page, or by using a specialized search string that’s really only formable by clicking links only from the tag profile. The result is an inconsistent search experience for users when it comes to tags.

Conclusion

I’m overwhelmingly optimistic about the social story in SharePoint 2013. It’s been missing for too long and has hindered adoption of SharePoint as a fundamental enterprise collaboration platform. For enterprises which have internet filters restricting access to Facebook, Twitter, and many other social networking sites, this will come as a welcome relief to employees who enjoy that form of social interaction. The communities and the newsfeed bridge two types of social communities: the forum community with the more mainstream social media community.

That said, Microsoft has introduced a major uphill battle for the enterprise to strategize on how to roll out social. I firmly believe that leaving it strictly to the community to self-describe their social story will result in the same cluttered mess that SharePoint is in most enterprises. Adoption will languish and the story gets that much more difficult for the organization to trust the social story in v.Next.

My gut recommendation for tagging in SharePoint 2013 is the following. This is likely to evolve as I come to understand the original intent behind how #tags were supposed to play with social tags.

• If you’re rolling this new, recommend users always define tags in terms of #tags.
• If you’re upgrading:
  • See which keywords make sense to resurface as #tags.
  • Delist the original keywords you converted to #tags as taggable.
• Drop the Tag Cloud Web Part into community sites to surface popular tags.
• Appoint someone in the organization as being responsible for the Keywords and Hashtags term stores.
  • This person should be responsible for pre-populating these term stores with common terminology throughout the organization.
  • This person should seek out other more specific terminology used within specialized departments.
  • This person should prune irrelevant, inappropriate, or duplicate terms in each of the stores to keep the sets fairly sane and easy for users.
• Give employees time to play with and learn to use the social features in SharePoint 2013.
  • Users who have used Facebook and Twitter understand the value and will quickly feel at-home.
  • Users who have used forums will also feel right at-home with communities and will quickly want to create their own.
    • Dissuade them from creating a new community unless they have a significant enough interest and make them realize they need to be a curator to keep it alive.
  • Users who are anti-social or have not adopted Facebook or Twitter (like my parents) need this time to understand the value of that feature.
  • All users need time to figure out what they have to do to follow #tags, users, documents, and lists.
  • Come up with a compelling story of how your department uses social.
    • After all, if you’re not eating your own dog food, how do you expect your users to use it?

Either way, the social features will really help to drive adoption of SharePoint 2013 and you’ll enjoy them when you start playing with them. If you have any questions, don’t heistate to drop a comment below or shoot me an e-mail at chris (at) chrisweldon (dot) net.

On October 22nd, I’ll be presenting with several of our best Azure consultants at the Windows Azure Developer Camp at Microsoft’s Houston Offices. Improving Enterprises has been working on our Azure competencies for the last couple of years and have had great success at several customer engagements where we deployed solutions to Azure. These engagements combined with the amazing talent we have here at Improving has allowed us to maintain our Microsoft Certified Partnership and become an Azure Circle Partner. In short, this should be an awesome free event.

At this event, we’ll be having not only presentations, but hands-on workshops on the following topics:

• Windows Azure Cloud Services
• Windows Azure SQL Database
• Windows Azure Virtual Machines
• Windows Azure Access Control Service
• Windows Azure Mobile Services

If you’ve always been interested in learning about how to make the move to the cloud, there’s never been a better time. This event is free, and everything (food, drinks, the software bits) are provided. It is BYOC (bring your own computer), so make sure to install the Azure Camps Kit ahead of the event! I look forward to seeing you there!

More Info and Registration

Disclaimer: Ultimately, I plan on coming back around and completely refactoring what you see below. It’s ugly as sin and can be much improved. I’ll write a separate blog post once I do. However, I felt it important to at least get this off my chest, as I’m notoriously bad about making sure to blog about solutions I find.

As a little more background, the customer had written a custom IHttpModule that sits in the IIS Pipeline for their SharePoint sites. The purpose of this module was to check if the user should be redirected to a different page based on the current requesting page. However, since this sits in front of a SharePoint site and serves 40k employees, it needed to be fast. Therefore, they used the following caching strategy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

public void OnBeginRequest(object sender, EventArgs e)
{
    if (HttpContext.Current.Cache.Count == 0)
    {
        DataTable dataTable = DataAccessLayer.GetUrlMap();
        foreach (var row in dataTable.Rows)
        {
            HttpContext.Current.Cache.Insert(row["SourceUrl"], row["DestinationUrl"]);
        }
    }

    string destinationUrl = HttpContext.Current.Cache[HttpContext.Current.Request.Url.ToString()] as string;
    if (destinationUrl != null)
    {
        // Redirect to destination Url.
    }
}

The following is a simple sample of the method being called in the DataAccessLayer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

public static DataTable GetUrlMap()
{
    string connectionString = ConfigHelper.GetConnectionString("URLMap");
    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        SqlCommand command = new SqlCommand("usp_GetUrlMap", connection);
        command.CommandType = CommandType.StoredProcedure;
        connection.Open();
        DataTable table = new DataTable();
        SqlDataAdapter adapter = new SqlDataAdapter(command);
        adapter.Fill(table);

        return table;
    }
}

(Disclaimer: this isn’t the real code. It’s a shortening of what’s actually happening.)

The Problem

Well, for the astute coder, you’ll quickly realize that there is no expiration set on the items inserted into the cache. Furthermore, there’s nothing else that actually invalidates the cache. Therefore, the only way that we can get new data into the cache is to perform an iisreset, something that is not allowed during normal business hours. Therefore, it was time to come up with a different solution for invalidating the cache.

We talked about options such as sending a specific query string parameter and using that as the “trigger” to empty the cache. However, that becomes problemmatic as we have a farm-based SharePoint scenario supported by a NetScalar load balancer, meaning we’d have to touch each web front end directly to ensure this works. That’s tedious and subject to yet another concern: the possibility that the data in the cache will differ between servers.

The Solution

This is when we started looking at automated mechanisms to “notify” the caching subsystem that data has been updated, and to invalidate the cache. This is where the SqlCacheDependency class comes into play. This class allows me to add a dependency to a specific database table for cache invalidation or a SqlCommand.

For applications written against Microsoft SQL Server 7.0 and SQL Server 2000, the SqlCacheDependency required configuration changes and some aspnet_regsql commands to be executed in order to prepare the enviornment to use the CacheDependency approprately. Even there, the SqlCacheDependency relied on polling the database - a rather expensive operation.

However, when using SQL Server 2005 and newer, Microsoft introduced a new asynchronous notifications system making it much simpler to know when changes to database records occur without having to constantly poll the table. This makes it much less expensive for tracking and something we were willing to consider. Even more importantly, it required no configuration changes to the Web Front Ends.

So, I started making the modifications to our caching engine to help support this. Because this is fairly tightly coupled to data access, though, my initial proof-of-concept includes some mandatory modifications to the data access layer (something I was not particularly comfortable with):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

public DataTable GetUrlMap()
{
    string connectionString = ConfigHelper.GetConnectionString("URLMap");
    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        SqlCommand command = new SqlCommand("usp_GetUrlMap", connection);
        command.CommandType = CommandType.StoredProcedure;
        connection.Open();

        SqlCacheDependency cacheDependency = null;
        try
        {
            cacheDependency = new SqlCacheDependency(command);
        }
        catch (DatabaseNotEnabledForNotificationException)
        {
            try
            {
                SqlCacheDependencyAdmin.EnableNotifications(connectionString);
            }
            catch (Exception)
            {
            }
        }
        catch (TableNotEnabledForNotificationException)
        {
            try
            {
                SqlCacheDependencyAdmin.EnableTableForNotifications(connectionString, "URLMap");
            }
            catch (Exception)
            {
            }
        }
        finally
        {
            DataTable table = new DataTable();
            SqlDataAdapter adapter = new SqlDataAdapter(command);
            adapter.Fill(table);

            HttpContext.Current.Cache.Insert("URLMap", table, cacheDependency);
            return table;
        }
    }
}

The above code is fairly straight-forward. The database and the table must have notifications setup at the SQL-level in order to have the caching dependency work in the prescribed manner. This is done through the SqlCacheDependencyAdmin class calls above. According to the MSDN documentation, the only time exceptions would occur when calling those methods would be when the user executing the commands does not have the required permissions.

Other than setting up the SqlCacheDependency, everything else is the same. Oh, except for the fact that you have to supply the dependency with the object you are storing in the cache. Therefore, now the data access layer is now strongly coupled to the caching mechanism. I feel really dirty doing this.

If you look at the original code, you’ll notice that I was caching each of the URLs as separate items in the cache. Unfortunately, if you attempt to do that here, you will get an exception with the message:

1

An attempt was made to reference a CacheDependency object from more than one cache entry.

Therefore, the “fix” for this is to store the entire DataTable in a single cache entry. This isn’t bad, but means we have to move our lookup for the matching redirection record from “hiding” it within the caching logic to being a little more explicit. I actually plan to refactor this and use something like a Dictionary to make lookups still be more performant than right now - O(n).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

public void OnBeginRequest(object sender, EventArgs e)
{
    DataTable dataTable = null;
    string destinationUrl = string.Empty;
    if (HttpContext.Current.Cache.Count == 0)
    {
        // Returning from the DataAccessLayer is redundant now.
        dataTable = DataAccessLayer.GetUrlMap();
    }

    dataTable = HttpContext.Current.Cache["URLMap"] as DataTable;
    foreach (DataRow row in dataTable.Rows)
    {
        if (row["SourceUrl"].Equals(HttpContext.Current.Request.Url.ToString(), StringComparison.InvariantCultureIgnoreCase))
        {
            destinationUrl = row["DestinationUrl"];
            break;
        }
    }

    if (!string.IsNullOrEmpty(destinationUrl))
    {
        // Redirect to destination Url.
    }
}

SQL Stored Procedure Gotcha

After all of the changes that I made, when I stepped thru the debugger, I was noticing very strange behavior. After I loaded items into the cache, the first request to the cache immediately afterwards (literally the following line after GetUrlMap()) was returning null - it couldn’t find the DataTable it just cached. Very pecular indeed.

I suspected it had something to do with the stored procedure. First, I read a couple of articles which indicated that you must make sure that the SQL command executed in the stored procedure reference all schemas and tables - explicitly. Furthermore, you could not select all columns using the wildcard (*). Instead, you had to be explict with the columns you select. So, the following two SQL commands would not work using the SqlCacheDependency:

1
2

SELECT * FROM URLMap;
SELECT SourceUrl, DestinationUrl FROM URLMap;

Instead, you had to change to be the following:

1

SELECT [dbo].[URLMap].SourceUrl, [dbo].[URLMap].DestinationUrl FROM [dbo].[URLMap];

Unfortunately, after making these changes, it didn’t solve the problem. I continued to hunt down the issue and managed to come across an ASP.Net forum post which suggested to look for the following in my stored procedure:

1

SET NOCOUNT ON

Sure enough, this was in my Stored Procedure (as it seems to be for most). After removing, the cache resumed working correctly and didn’t immediately purge after inserting. Essentially, the MSDN Page for Creating a Query for Notification indicates that using SET NOCOUNT ON is not allowed for stored procedures.

Specifics with SharePoint

I came across two specific issues that you have to keep in mind with loading this in SharePoint. First is how you declare the SqlCacheDependency. The reference documentation indicates to use just the name of the database and the name of the table. This is fine if you have the connectionstring stored in your Web.config file. If you don’t (as is the case in our environment), then you cannot use this manner of watching for database table changes - you must use a stored procedure or regular SQL Command.

Second, you have to be mindful of the fact that you cannot simply drop the SqlDependency.Start(connectionString) in Application_Start - SharePoint is in strict control of Application_Start However, putting it in your IHttpModule’s Init() method is totally fine:

1
2
3
4
5
6

public void Init(HttpApplication application)
{
    string connectionString = ConfigHelper.GetConnectionString("URLMap");
    SqlDependency.Start(connectionString);
    application.BeginRequest += OnBeginRequest;
}

Happy coding!

The last couple of weeks in September I was tasked with identifying and fixing a problem with our production FAST Search for SharePoint environment: slow crawl rates. Our incremental crawls were taking anywhere between 45 minutes and several hours to run for each cycle, and a full crawl was taking in excess of 65 hours to crawl just under 1 million documents. Atrocious. I had little previous experience administering FAST search environments, and by the end of this process, I feel like a solid power administrator (but still by no means an “expert”). This post is meant to share my thought process, findings, resources, and other information available to me to help identify the bottlenecks and implement a fix to this problem.

About FAST Search for SharePoint (FS4SP)

FAST Search was originally developed by a company called FAST Search and Transfer and was acquired by Microsoft in 2008. The acquisition was a strategic move on Microsoft’s part to enter the enterprise search market. Although Microsoft had their own search products (Microsoft Search Server and Search Server 2010), the scalability and feature sets provided with FAST Search far outweigh that of Microsoft’s search products. Unfortunately, these extra features come at a premium, but are truly necessary if you really want a rich search experience for users.

Architecturally, FAST search components are very easy to configure and scale out to increase not only query thruput, but also crawling and indexing thruput and capacity. This is done by using a matrix configuration where servers are arranged into rows and columns. When you read the technical documentation, by adding rows, you increase redundancy of services, but also query capacity. However, when you add columns, you increase indexing capacity and thruput. Discussing the nuances of search rows and columns is another blog post in itself, so I’ll leave that for a future post.

Beyond simply adding servers to rows and columns, deciding which servers host which services involved in the content pipeline is extremely easy. All it takes is a configuration file change and issuing a single command across all servers in the FAST farm. You can also have multiple instances of the services to provide load balancing and high availability. How this works is also worthy of another blog post.

The following is how our environment is architected:

This is a fairly typical environment, where we are taking advantage of two rows of indexers for high availability, with two index columns for expanded capacity. Document processing is spread out over all servers to have more than enough pipeline bandwidth to constantly work on the large volume of content that our crawlers will be sending to our environment. Finally, these servers have 12 cores, 32G of RAM, and two GigE nics in a teaming configuration: more than beefy enough servers to quickly chew through the massive amount of content expected to be thrown its way.

The Problem

Our crawl rate was atrocious. The last full crawl took 63:47:23 to crawl 870K+ items - an effective crawl rate of 3.78 documents per second. Incremental crawls were taking in excess of 20 minutes each. This didn’t make much sense given the horsepower that we had in our FAST and SharePoint environment. So, I was tasked with trying to get this crawl rate improved. But first, I had to learn how to administer a FAST environment.

Brain Food - The Books I Used

One of the first books that I picked up was Professional Microsoft Search: FAST Search, SharePoint Search, and Search Server (ISBN 978-0-470-58466-8). I read the majority of the book, but eventually stopped. This book is not very technically-oriented. It gives a great overview of the differences between the OOB SharePoint Search, Microsoft Search Server, and FAST Search. It provides screenshots and basic information on how to install FAST. It also provides the theory behind capacity planning and architecture, and considerations when building search-based applications. Ultimately, however, tips and information on how to administer the environment and identify bottlenecks was woefully inadequate.

The next book I started to read was Working with Microsoft FAST Search Server 2010 for SharePoint. After cracking open this book, I was much more pleased and found more information on the various powershell commands and executables provided by FAST. In fact, one of the most important resources was a series of links I was able to find throughout the book that point to different pages on TechNet. These pages include the tools and other detailed information necessary to administer the FAST environment:

• FAST Search PowerShell Commandlets
• FAST Search for SharePoint Command-Line Tools
• Performance Counters
• Performance and Capacity Monitoring

One of the best resources I found was on YouTube, titled Troubleshooting FAST Search and the SharePoint Crawler - An Operational Viewpoint by Kristopher Loranger from Microsoft. One of the best tidbits in the video was this: a good crawl rate should be between 45 and 60 documents per second. So, I had a goal to shoot for. But, most importantly, he gave a simple trick to help identify where the bottleneck is: temporarily suspend indexing. By suspending indexing and performing another full crawl, you can identify whether the problem is on the SharePoint infrastructure, or on the FAST infrastructure side. If there is speedup after suspending indexing, the problem is FAST. Otherwise, the problem is SharePoint. A pretty simple technique to keep in your back pocket.

The best article I found on the TechNet site was that on Performance and Capacity Monitoring. This told me specifically which performance counters to monitor, and how to use that information to identify bottlenecks. I kept referring back to this article when setting up Performance Monitor on each of the various servers.

Interesting FAST Search Log Entries

While looking through and gathering information that could be used to identify any bottlenecks in our environment, I decided to look at the FAST admin server logs in <FASTInstall>\var\log\all.log. In there, I noticed a significant number of the following entries:

1
2
3
4
5
6

systemmsg: Subsystem operation call timeout (session 3, ops 558->558), sleeping 84 seconds before resubmitting operation set
systemmsg: Subsystem operation call timeout (session 3, ops 551->551), sleeping 47 seconds before resubmitting operation set
systemmsg: Subsystem operation call timeout occured 3 times, sending callback
systemmsg: Subsystem operation call timeout occured 3 times, sending callback
systemmsg: Subsystem operation call timeout occured 3 times, sending callback
systemmsg: Subsystem operation call timeout occured 3 times, sending callback

This was a significant problem during the full crawl. In fact, 15MB of those log entries kind of a problem. It was happening roughly once a second. Ultimately, what confused me about those log entries was the fact they were logged at an INFO log level - not warning or error. Call timeouts that occur that frequently should indicate some significant problem - but where? Was it a network problem, a processing problem, or some configuration problem? My gut reaction was to consider this a network problem, but I needed more statistics to be able to more acutely identify the elements influencing this problem.

Performance Monitor - Knowledge is Power

As mentioned above, I kept referring back to the Performance and Capacity Monitoring document on Microsoft TechNet and proceeded to setup monitors in Performance Monitor. The following explains what I setup on each of the servers in the environment.

• Content SSA / Crawler
  • Batches Ready (OSS Search FAST Content Plugin)
  • Batches Submitted (OSS Search FAST Content Plugin)
  • Processor % (Processor)
  • Bytes Total/sec (Network Interface)
• Content Distributors
  • Document Processors (FAST Search Content Distributor)
  • Document Processors Busy (FAST Search Content Distributor)
  • Average Dispatch Time (ms) (FAST Search Content Distributor)
  • Bytes Total/sec (Network Interface)
• Indexers
  • Current Queue Size (FAST Search Indexer Status)
  • FiXML Fill Rate (FAST Search Indexer)
  • Active documents (FAST Search Indexer Partition)
  • Bytes Total/sec (Network Interface)

Understand the Data Flow before Understanding the Statistics

Before we began to look at the statistics, it’s important to understand how the content flows from end-to-end. A problem earlier in the content pipeline can manifest bad / inappropriate metrics further down the line. So, identifying the bottleneck where it really is (as early as possible) is truly important. The steps of content processing is best represented by this image found on TechNet:

Let’s take a look at a few of the metrics I managed to gleam from these performance metrics.

Content Search Service Application / Crawler

Per our architecture, we have only a single crawler component. So, getting a sense of how much content the Content SSA is pushing to the Content Distributors is the first step in the process. Again, if you refer back to the diagram of the content processing pipeline, sending data from the crawler to the document processor is the first step. So, let’s look at the network diagrams for this server:

We are looking at two network interfaces as there is not a consolidated performance counter for the “virtual” paired interface that Broadcom creates for us. Therefore, I have to look at each of the network interfaces. Nevertheless, you can see that the peak network traffic for this server was roughly 57 MBps, averaging at only 443 KBps. This isn’t really high thruput considering the dual paired GigE network interfaces on this server.

If we look at the network traffic on the admin server, it echos that of the Content SSA - low network thruput (peak traffic 127 MBps, averaging around 477 KBps):

So, while this is looking like a network bottleneck, it’s still important for us to look at the other statistics. Unfortunately, I don’t have screenshots of the next important statistic: Batches Ready. I found that when looking at that statistic, it rose sharply and remained high throughout the crawl - in excess of 15K items, regularly refilling after dropping to about 7-8K. What this meant was the crawler was able to chew through data faster than the FAST search environment was able to process it. If Batches Ready was close to or at 0, then that would mean the crawler could not keep up with the FAST search processing. So, this news made me question whether I was on the proper path hunting down a network issue.

However, this did not rule out the possibility of a network problem. I was able to look at the Batches Submitted statistic and find that it was very low. This is indicative of some bottleneck (either network or disk), as the content server should have been able to push all of the batches as soon as they were ready. But, the amount being submitted during each cycle was dismal - less than 100 batches per cycle.

So, let’s switch over to the Document Dispatcher to figure out what may be going on there. Again, continuing with my hunch, I wanted to look at the Average dispatch time. This is solidly a network bottleneck monitor, and the results below further my suspicions. Average dispatch times were in excess of 12,900 ms. According to Microsoft, dispatch times over 10 ms indicate network congestion.

Of course, an obligatory look at the document processors in-use is necessary. Disappointingly, we were not using even half of our document processors on average. But, this is likely because we can’t feed enough content fast enough to the servers, leaving the document processor threads idle, waiting for work. An easy way of double checking the values below is to run the following command from command line:

1

psctrl status

This will show the status of all document processors in the farm. Based on the graph below, most times when I was checking that, most DPs were idle. However, just because we had a little under half working at seemingly all times, they weren’t doing much. This metric is literally either an indicator that a DP is working or it’s not. If you dig into the workload by using the command above, you’d have found that most servers were only processing between 4 and 5 documents at a time. Looking at the CPU utilization of each of the document processing servers, I was barely exceeding 30% usage because of this lack of processing.

As I continued to look at the other metrics (like processor, network, disk), it became increasingly apparent that those metrics were deficient due to the lack of sufficient content reaching the servers. I continued to search the internet for recommendations, and finally managed to come across…

The Solution

The solution actually came in the form of two Microsoft articles: one on TechNet, the other on the Microsoft KnowledgeBase site. I finally managed to walk down the hierarchy of articles on the FAST Search Server 2010 for SharePoint documentation to the Troubleshooting section. Inside, there was an article titled Crawling: Slow crawling and slow copying between the indexer and the search nodes. This indeed identified the problems that I surmised: the network was the bottleneck.

In reading about the solution, it recommended that I turn off Network Task Offloading. This was down at a hardware level that I had never really learned, despite my systems administration experience. In essence, Network Task Offloading helps ensure the responsibilities of initiating and validating network traffic between systems is delegated to the network controller hardware rather than the CPU. It makes sense: I want specialized controllers (network controllers, audio controllers, video controllers) to be responsible for processing the data they were specifically created to process. However, as the TechNet article mentions, Task Offloading causing slow crawl problems is typically seen in virtualized environments, and for good reason. In a virtual environment, the VM does not have direct access to the network controller. Instead, it’s provided with a virtual adapter. If the virtualization kernel doesn’t know to translate requests to that virtual adapter to the physical network controller, then inevitably the CPU gets the responsibility of processing those requests - at a significant premium over just telling the CPU to do it in the first place.

However, that was only part of the solution. When I continued to search, I came across a Microsoft KnowledgeBase article titled You cannot crawl or index SharePoint sites in FAST Search Server 2010. This article referenced not only Network Task Offloading, but also TCP Chimney Offload. Ultimately, I loved that the KnowledgeBase article explained why these two things were bad for the FAST Search environment: IPSEC. IPSEC is implemented at Layer 3 (Networking Layer) of the OSI model. As a result, having TCP connections offloaded to hardware, unfamiliar with the security policies in place, will result in improper network activity.

Though the solution was straight-forward, it took me about 10 minutes per box to impelment. Do the following steps to all of your FAST servers as well as your Content SSA servers (crawlers):

Disable TCP Chimney Offload and Task Offload

Open a command prompt on your server as an administrator (make sure to use Run as Administrator if UAC is in effect). Then, run the following commands:

1
2

netsh int tcp set global chimney=disabled
netsh int ip set global taskoffload=disabled

You can then verify those are disabled with the following commands:

1
2

netsh int tcp show global
netsh int ipv4 show global

Disable Checksum and Large Send Offload at Hardware Level (for Physical Servers)

If you are not using Virtual Machines, you need to check the hardware-level configuration of each of your network adapters. In my case, we were using a Dell Server complete with Broadcom GigE network adapters which had to be configured through it’s proprietary Broadcom Advanced Control Suite 3. I went through each of the network adapters in that interface and found the following settings:

Then, I had to check the server-level configuration for Checksum Offload and ensure it was set to disabled:

Finally, our systems administrators setup teaming across our network adapters for high availability and load balancing. I double checked the teaming configuration that IPv4 Checksum Offload and IPv4 Large Send Offload were not enabled. Thankfully, once I disabled those features on each of the network cards, the software was intelligent to disable that at the teaming level. However, it would be a good idea that you double check to make sure that your network card does the same:

Reboot the Server

This one is patently obvious. Once you muck with these network settings, you need to reboot to make sure they take effect.

Results

After making all of these changes, I started another full crawl to see if that made any bit of difference. At the completion of the full crawl, a total of 936872 results in 3:28:47 for an effective rate of 74.788 documents per second. This is an increase of nearly 20 fold, an amazing improvement, and well exceeding the desired crawl rate. To look at the results, let’s start from the beginning and look at the networking charts on the crawl server:

In looking at the Content SSA updates, the maximum thruput was 210 MBps, and the average was 8 MBps! A great speedup. That’s a sufficient level of network traffic, so let’s check out the admin network traffic.

While the graph is less than menacing, the numbers don’t lie: maximum thruput of 194 MBps and an average thruput of 10.9 MBps. Also significant speedup. Knowing that the content over the network seems to be flowing well, let’s see how the crawl server’s queue looked:

The peak was still higher than I would like, but still much improved compared to previous: 9300. Considering the average was 922 batches ready for dispatch, I’m more than happy with that level of improvement. Let’s check out our document processor utilization:

You’ll notice we have the same number of document processors available as previously: 37. However, what you’ll also notice is at several points we had all document processors busy - none were idling. This typically occured as the processing load increased, and died off as the document processors completed processing. To verify that we were feeding content quickly enough to do the document processors, let’s take a look at our dispatch rate:

The average dispatch times were much lower. The peak was still unacceptable (at 8190ms), but you can see that was a rather odd fluke in our measurements. There may have been some other blocking process or anomaly in the network to cause this dispatch time, as the overall average is significantly lower than the max: 213ms. This is still outside of the recommended range Microsoft states (10ms), but is still significantly lower than our previous result.

Ultimately, the last thing to check was how the index servers responded to these updates. If we look at the processor metrics alone, you can see that the processor was finally being more fully utilized. Although these performance graphs barely show 100% utilization, the live graphs were almost peaking at 100% constantly:

But although the processor is a good metric of how utilized the server is, what’s more important to determine that the indexers have sufficient amount of data to work against, is the FiXML rate. When we look at that, we see the rate is largely above 75%, the threshhold Microsoft recommends is necessary for a “healthy” FAST indexing envrionment:

Looking at our crawl times now, I’m much happier to see incremental crawls taking less than 10 minutes a piece, even when indexing 40K+ items:

Admin Reports and Conclusion

In short, troubleshooting this issue was no walk in the park. FAST is not an easy system to understand. Throwing SharePoint into the mix (which is just as large and difficult) only makes matters worse. However, by paying close attention to the details and studying how your system is behaving, you’ll eventually find what you’re looking for.

<rant> Finding the information on what to look for, though, can be just as challenging. I found that things really started to take off for me only after I started reading the Microsoft FAST book. TechNet had the required documentation that I needed to understand how to diagnose the problem. Yet, I found that finding information on TechNet was extremely challenging. For a product that can cost in the tens of thousands of dollars for just a small environment, I’d really like to see Microsoft invest some time in really understanding how end users look through their documentation, and really improve it. The “auto-hiding” links in the navigation heiarchy on the left of the page suck. What sucks even more? Not having an automatically expandable hierarchy so I can browse the hierarchy of contents. The fact that it would take me 15-20 clicks to get to the bottom of the FAST documentation is absurd - I should be able to have a full ToC - not a ToC at every major section all the way to the bottom of the stack. </rant>

I leave you with these wonderful Administrative Report graphs, just because I think they’re actually really useful bits of information, though I don’t feel like diving in to explain all of these.

If you get this, it’s time to crack open the Event Viewer and the SharePoint Logs. If you’re not on a single server environment (aka: a Farm), trying to find the Correllation ID is going to be truly fun. However, once you’ve found the Correllation ID, look for the next Unexpected or Critical error. Initially, I came across the following Unexpected error:

1
2
3

CoreResultsWebPart::OnInit: Exception initializing: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e)

with the detailed exception being:

1
2
3
4
5

Internal server error exception: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e) System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e)

However, when I went digging a little further, I found the following error. If you find it as well, you’re on the same track that I am:

1

Failed to connect to FASTSEARCHSERVER:PORT Failed to initialize session with document engine: Unable to resolve Contentdistributor

FASTSEARCHSERVER:PORT is the server and port of the FAST Search Service you should have already installed. If you find this, your query service application may not be able to talk to the FAST Search Server. I suggest you check for one (or all) of the following.

Ensure the FAST Search Services are Running

First, log in to all to all servers where you have installed the FAST Search Services. You’ll need to open the Services Management console by doing the following:

1. Go to Start > Run
2. Type services.msc and press enter
3. If you have UAC, you’ll be prompted for your Admin username and password

In the list of services, search for the following and make sure all are started:

• FAST Search for SharePoint
• FAST Search for SharePoint Monitoring
• FAST Search for SharePoint Sam Admin
• FAST Search for SharePoint Sam Worker

If any of them are not started, I suggest shutting them all down. From there, only start the FAST Search for SharePoint service. This should start the other three services as well. Once they are all started, perform an iisreset on the web front ends (especially the one running the SharePoint Search Query and Site Settings Service) and try the search again.

SSL Certificate Problem

If the search is still failing after doing the above check, you need to check to see if the SSL Certificate used for FAST Search communication has expired. I found a great TechNet article (which points to an even better Blog Post by SharePoint Malarkey regarding this problem. Review the instructions in the blog post to verify whether you indeed have this problem.

Is Anyone the Query Server?

If you continue to receive errors after performing the above steps, check your SharePoint logs again for the Correllation ID. Somewhere around there you may find the following Unexpected error that looks similar to that at the top of this blog post:

1

CoreResultsWebPart::OnInit: Exception initializing: Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application.

If you look above this error, you may find another Critical error:

1

There are no instances of the Microsoft.Office.Server.Search.Administration.SearchQueryAndSiteSettingsServices started on any server in this farm.

If you indeed see these exceptions, then you need to check this out. Do the following:

1. Go to Central Administration
2. Go to Application Management > Manage Services on Server
3. Go through each of the Servers and see if any of them have the Search Query and Site Settings Service started.
4. If none have this service started, start this service on the same server that’s running SharePoint Server Search according to the Servers in Farm page.
5. Run iisreset on the server where you found the Correllation ID.

By this point, things should start looking up and you should actually get a search page with results.

I attended Austin Code Camp a few weekends ago and had a blast! There were plenty of enthusiastic, passionate individuals gathered and ready to learn - the makings for a great one day conference. While there, I attended the session Must-Know Design Patterns by Rob Vettor. In that session, he discussed several very useful patterns, including the Decorator Pattern, the Factory Pattern, and numerous other useful patterns. In seeing his code samples for the factory pattern, I was a bit disappointed to see his entire block of if/else if/else statements simply moved from his business logic to the factory. While this is still technically the better approach than leaving it in business logic, I find there’s a better way to do this. In fact, there’s a zero case statement way of doing this in C#. If you’re curious to find out how to create a factory without a single if or switch statement, read on.

If you’ve never heard of it, check out the Anti If Campaign. I wasn’t aware of this until about a year ago, but prior to that I was already a practitioner of their preachings. The goal is simple: get rid of most (if not all) case statements in your code. Why? An increase in the number of case statements increases the branch (cyclomatic) complexity of your logic. You have to add more unit tests to ensure you hit everyone of those branches. It’s far too often the case that your code isn’t just a set of conditional statements - it has other logic surrounding it. It’s messy, and it’s poor design.

If you are truly paying attention to object-oriented design patterns, you’d realize there are ways for you to structure your code in a manner that minimizes or eliminates the need for conditional statements. Most people revolt when I make this assertion. Some of the arguments I’ve heard include:

1. Won’t you create too many classes to maintain? I have a hard enough time navigating my codebase as-is.
2. Aren’t classes “heavy”? Won’t that bloat the application?
3. How do I know what my application is doing without if/else if/else statements?

My responses are usually:

1. You’ll often create many more classes. But, if you have a) sound organizational plans and b) a somewhat-modern IDE you can find any of the classes you use. Learn to use your IDE so you can help you navigate your codebase.
2. It depends on the class. Often, I create many extremely lightweight classes and only allow them to live while I need them to help keep memory consumption to a minimum.
3. Are you flinging code at your screen like a monkey flings poo at others? How else do you expect to build your application? Understanding these (and other) design principles will allow you to make sense of what your app is doing, rather than treating it like some voodoo blackbox.

The Problem

Let’s take a look at a sample problem that I’d like to make much simpler. The code sample below should be fairly easy to follow. We are a custom factory responsible for building different makes and models of vehicles. As you can see, the majority of our business logic is tied up trying to figure out how to create the vehicle and assign it the options.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

public class VehicleController
{
    public ActionResult buildVehicle(string make, string model, string color, int numDoors, string[] options = string[0])
    {
        Vehicle vehicle = null;
        if (make.Equals("Honda")) {
            if (model.Equals("Civic")) {
                // Ooh! My favorite!
                vehicle = new Civic();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("Accord")) {
                // Only comes in 4 doors. So, ignore numDoors.
                vehicle = new Accord();
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("CR-V")) {
                // Only comes in 4 doors. So, ignore numDoors.
                vehicle = new CRV();
                vehicle.Color = color;
                vehicle.Options = options;
            } else {
                throw new InvalidArgumentException("Cannot make requested model for Honda");
            }
        } else if (make.Equals("Chevrolet")) {
            if (model.Equals("Suburban")) {
                vehicle = new Suburban();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("Silverado")) {
                vehicle = new Silverado();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else {
                throw new InvalidArgumentException("Cannot make requested model for Chevrolet");
            }
        } else if (make.Equals("BMW")) {
            // Pompus luxury vehicle.
        } else if (make.Equals("Ferrari")) {
            // Not likely in my lifetime.
        } else if (make.Equals("Lamborghini")) {
            // I wish.
        }

        // Save.
        this.vehicleContext.Attach(vehicle);
        this.vehicleContext.SaveAll();

        // Display our newly created vehicle.
        return View("BuildVehicle", vehicle);
    }
}

The cyclomatic complexity of this action alone is ridiculous. Controller actions should be simple and straight-forward: collect data and validate, pass to business logic, return response. Any controller action over 20 LoC is doing too much and some refactoring should be considered. So, what should we do?

The Pattern

Let’s take the first step and refactor that gigantic if block into a separate factory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

public class VehicleFactory
{
    public Vehicle buildVehicle(string make, string model, string color, int numDoors, string[] options = string[0])
    {
        Vehicle vehicle = null;
        if (make.Equals("Honda")) {
            if (model.Equals("Civic")) {
                // Ooh! My favorite!
                vehicle = new Civic();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("Accord")) {
                // Only comes in 4 doors. So, ignore numDoors.
                vehicle = new Accord();
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("CR-V")) {
                // Only comes in 4 doors. So, ignore numDoors.
                vehicle = new CRV();
                vehicle.Color = color;
                vehicle.Options = options;
            } else {
                throw new InvalidArgumentException("Cannot make requested model for Honda");
            }
        } else if (make.Equals("Chevrolet")) {
            if (model.Equals("Suburban")) {
                vehicle = new Suburban();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else if (model.Equals("Silverado")) {
                vehicle = new Silverado();
                vehicle.Doors = numDoors;
                vehicle.Color = color;
                vehicle.Options = options;
            } else {
                throw new InvalidArgumentException("Cannot make requested model for Chevrolet");
            }
        } else if (make.Equals("BMW")) {
            // Pompus luxury vehicle.
        } else if (make.Equals("Ferrari")) {
            // Not likely in my lifetime.
        } else if (make.Equals("Lamborghini")) {
            // I wish.
        }

        return vehicle;
    }
}

This greatly simplifies our VehicleController (see below), but does little to solve our cyclomatic complexity problem. Furthermore, one of the biggest problems I have with this as a factory is it will have to be modified every time a business use case related to the factory changes. We need a better solution to help make this easier to maintain and unit test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

public class VehicleController
{
    public ActionResult buildVehicle(string make, string model, string color, int numDoors, string[] options = string[0])
    {
        VehicleFactory factory = new VehicleFactory();
        Vehicle vehicle = factory.buildVehicle(make, model, color, numDoors, options);

        // Save.
        this.vehicleContext.Attach(vehicle);
        this.vehicleContext.SaveAll();

        // Display our newly created vehicle.
        return View("BuildVehicle", vehicle);
    }
}

So, the first option I’d push for is moving the actual creation of each make into a separate factory. So, we’ll have a HondaFactory, a ChevroletFactory, and other factories for each of our other types. So, looking at the refactor for the HondaFactory, we have something similar to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

public class HondaFactory
{
    public Vehicle CreateVehicle(string model, string color, int numDoors, string[] options = string[0])
    {
        Vehicle vehicle = null;
        if (model.Equals("Civic")) {
            // Ooh! My favorite!
            vehicle = new Civic();
            vehicle.Doors = numDoors;
            vehicle.Color = color;
            vehicle.Options = options;
        } else if (model.Equals("Accord")) {
            // Only comes in 4 doors. So, ignore numDoors.
            vehicle = new Accord();
            vehicle.Color = color;
            vehicle.Options = options;
        } else if (model.Equals("CR-V")) {
            // Only comes in 4 doors. So, ignore numDoors.
            vehicle = new CRV();
            vehicle.Color = color;
            vehicle.Options = options;
        } else {
            throw new InvalidArgumentException("Cannot make requested model for Honda");
        }

        return vehicle;
    }
}

And here’s our ChevroletFactory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public class ChevroletFactory
{
    public Vehicle CreateVehicle(string model, string color, int numDoors, string[] options = string[0])
    {
        Vehicle vehicle = null;
        if (model.Equals("Suburban")) {
            vehicle = new Suburban();
            vehicle.Doors = numDoors;
            vehicle.Color = color;
            vehicle.Options = options;
        } else if (model.Equals("Silverado")) {
                vehicle = new Silverado();
            vehicle.Doors = numDoors;
            vehicle.Color = color;
            vehicle.Options = options;
        } else {
            throw new InvalidArgumentException("Cannot make requested model for Chevrolet");
        }

        return vehicle;
    }
}

Now we need to figure out how to get our main VehicleFactory to route requests to each of these different factories without conditionals. But, before we do this, let’s think about what we just did. We need to realize that each of the factories we created are doing the same thing. When you start having several classes doing the same thing, it’s time to introduce an interface. With this, we also introduce what will be termed as the evaluator:

1
2
3
4
5

public interface IVehicleFactory
{
    Vehicle CreateVehicle(string model, string color, int numDoors, string[] options = string[0]);
    bool CanCreateMake(string make);
}

CanCreateMake evaluates the make and returns whether this factory can handle it. We then are able to change our HondaFactory to look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

public class HondaFactory : IVehicleFactory
{
    public Vehicle CreateVehicle(string model, string color, int numDoors, string[] options = string[0])
    {
        Vehicle vehicle = null;
        if (model.Equals("Civic")) {
            // Ooh! My favorite!
            vehicle = new Civic();
            vehicle.Doors = numDoors;
            vehicle.Color = color;
            vehicle.Options = options;
        } else if (model.Equals("Accord")) {
            // Only comes in 4 doors. So, ignore numDoors.
            vehicle = new Accord();
            vehicle.Color = color;
            vehicle.Options = options;
        } else if (model.Equals("CR-V")) {
            // Only comes in 4 doors. So, ignore numDoors.
            vehicle = new CRV();
            vehicle.Color = color;
            vehicle.Options = options;
        } else {
            throw new InvalidArgumentException("Cannot make requested model for Honda");
        }

        return vehicle;
    }

    public bool CanCreateMake(string make)
    {
        return make.Equals("Honda");
    }
}

So, let’s store all of our vehicle factories into a collection in our primary VehicleFactory and pull out the one we want to use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

public class VehicleFactory
{
    private IList<IVehicleFactory> factories = new List<IVehicleFactory>();

    public VehicleFactory()
    {
        this.factories.Add(new HondaFactory());
        this.factories.Add(new ChevroletFactory());
    }

    public Vehicle buildVehicle(string make, string model, string color, int numDoors, string[] options = string[0])
    {
        IVehicleFactory usableFactory = this.factories.SingleOrDefault(fac => fac.CanCreateMake(make));
        if (usableFactory == null) throw new InvalidArgumentException("Cannot create vehicle of the requested make.");
        return usableFactory.CreateVehicle(model, color, numDoors, options);
    }
}

We’re definitely getting there, but there’s one glaring problem. As mentioned previously, I hate the way factories are usually implemented, as they require modification if we need to add a new conditional. We’re still in the same boat here. If we begin to manufacture a new make, we have to create that specific make’s factory and modify this central factory to register it. This breaks the Open/Closed Principle, and we need to fix it. Thankfully, our interfaces help.

At Lone Star PHP an individual approached me after my SOLID talk to mention that he’s not sold on the use of interfaces. He said that interfaces seem to be there only for the developers - as a means to help them add new classes that perform the same types of activities. But, he still didn’t see a computer-use for them. That’s when I busted out the use of Reflection. Reflection helps me walk my codebase and find objects of specific types without ever being aware of what is out there. Reflection will help us solve this problem.

1
2
3
4
5
6

public VehicleFactory()
{
    var discoveredFactories = Assembly.GetExecutingAssembly().GetTypes()
        .Where(type => type.GetInterfaces().Any(intrfce => intrfce.Equals(typeof(IVehicleFactory))));
    this.factories.AddRange(discoveredFactories);
}

This is now completely implementation agnostic and accomplishes our second goal: create a factory that’s safe against modifications - this factory is not likely to ever change. The added benefit that I’ve introduced is the ability to drop in any new make factory that implements IVehicleFactory, recompile, and deploy - and it will automatically be picked up by my assembly. That’s sexy.

Our final factory that all of our core business logic will use looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

public class VehicleFactory
{
    private IList<IVehicleFactory> factories = new List<IVehicleFactory>();

    public VehicleFactory()
    {
        var discoveredFactories = Assembly.GetExecutingAssembly().GetTypes()
            .Where(type => type.GetInterfaces().Any(intrfce => intrfce.Equals(typeof(IVehicleFactory))));
        this.factories.AddRange(discoveredFactories);
    }

    public Vehicle buildVehicle(string make, string model, string color, int numDoors, string[] options = string[0])
    {
        IVehicleFactory usableFactory = this.factories.SingleOrDefault(fac => fac.CanCreateMake(make));
        if (usableFactory == null) throw new InvalidArgumentException("Cannot create vehicle of the requested make.");
        return usableFactory.CreateVehicle(model, color, numDoors, options);
    }
}

Rinse and Repeat

Now that the basics have made themselves apparent, you should be able to figure out how to get rid of the remainder of the if/else blocks of this code. If you need further guidance, keep reading. Otherwise, I suggest you jump to the summary.

To get rid of the remainder of the if/else statements we have in our individual make factories, we apply the same principles that we did the first time around. We need yet another factory responsible for each model of the car. We then need a way to determine if that factory can handle the creation of the requested model. It’s time for yet another interface:

1
2
3
4
5

public interface IMakeVehicles
{
    Vehicle CreateVehicle(string color, int numDoors, string[] options = string[0]);
    bool CanCreateModel(string model);
}

So, let’s create our first vehicle, a Civic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public class CivicFactory : IMakeVehicles
{
    public Vehicle CreateVehicle(string color, int numDoors, string[] options = string[0])
    {
        Civic vehicle = new Civic();
        vehicle.Doors = numDoors;
        vehicle.Color = color;
        vehicle.Options = options;
        return vehicle;
    }

    public bool CanCreateModel(string model)
    {
        return model.Equals("Civic");
    }
}

We do the same thing for all of our other models. The conditional evaluator is moved from the if block into the CanCreateModel method, and the body of the if statement is moved into CreateVehicle. All that’s left is to apply the same trick that we did in the VehicleFactory to auto-discover our model factories. But - how do we make sure the Honda Factory only discovers factories relating to it, rather than all makes? The simplest approach here is to refactor IMakeVehicles to be a generic interface:

1
2
3
4
5

public interface IMakeVehicles<TModelType> where TModelType : Vehicle
{
    TModelType CreateVehicle(string color, int numDoors, string[] options = string[0]);
    bool CanCreateModel(string model);
}

We then update our CivicFactory to use the following definition:

1
2
3
4

public class CivicFactory : IMakeVehicles<Honda>
{
    public Honda CreateVehicle(...);
}

This allows us to use the same Assembly trick from earlier on a strongly-typed interface. The assumption here is that Civic : Honda and Honda : Vehicle. Without that relationship, this trick will not work.

We finish up by changing our HondaFactory to look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

public class HondaFactory : IVehicleFactory
{
    private IList<IMakeVehicles<Honda>> factories = new List<IMakeVehicles<Honda>>();

    public VehicleFactory()
    {
        var discoveredFactories = Assembly.GetExecutingAssembly().GetTypes()
            .Where(type => type.GetInterfaces().Any(intrfce => intrfce.Equals(typeof(IVehicleFactory<Honda>))));
        this.factories.AddRange(discoveredFactories);
    }

    public Vehicle buildVehicle(string model, string color, int numDoors, string[] options = string[0])
    {
        IVehicleFactory usableFactory = this.factories.SingleOrDefault(fac => fac.CanCreateModel(model));
        if (usableFactory == null) throw new InvalidArgumentException("Cannot create vehicle of the requested model.");
        return usableFactory.CreateVehicle(color, numDoors, options);
    }

    public bool CanCreateMake(string make)
    {
        return make.Equals("Honda");
    }
}

Summary

As we have seen, you can easily refactor your logic in such a way that’s a lot more straight-forward to maintain. You’ve created a factory that can be reused in multiple places across your application. Instead of having to maintain a gargantuan and growing class, you being to worry about micro-factories that are focused on your specific needs. Your micro-factories are able to be reused across components. You have encapsulated all logic associated with the creation of a particular vehicle in a single class, rather than splitting the determination of when to create the class and how to create it in separate classes. Your unit test classes for each of these components are so small, you stop worrying about how to create your 18-level chain mock to test a single conditional statement. In short: you achieve OO bliss.

What I’ve introduced is not new. In fact, this is a variation of the Command Pattern. I also heavily rely on this pattern when I’m implementing strategies within my code. Strategies change often, so when I need to introduce a new strategy, I create the class, implement the interface, rebuild, and go! Removing a strategy is as simple as pressing the delete key. One class and everything else is taken care of for me.

If you have questions or comments about this pattern, please feel free to reach out to me via e-mail or through the comment page below. I would love to hear other’s ideas on this implementation.

Over this past weekend, I spent time working on a new set of web services for an existing customer. Previously, their web services were XMLRPC based, and performed authentication by sending the username and an API key rather than a password over the wire. These web services were secured via HTTPS, but use of HTTPS was not mandatory. Since the API exposed methods that allowed you to create, read, and delete records associated with a customer, the use of this API without using SSL could be disastrous for its customers.

We had multiple outstanding requests to convert our web services to be REST-based and have also had the request to implement OAuth. These are definitely the direction we are heading for the project, but due to resource contraints (i.e. myself being the only developer), I have to be mindful about when I can deliver value. While implementing the new RESTful services this weekend, I finally came upon the authentication requirement which caused me to question: what is the most appropriate mechanism for authenticating my identities? Read on for the justification of what authentication mechanism I decided upon and how I implemented it in Zend Framework.

Possible Authentication Solutions

The business model our customer uses involves many API customer integrations. While the majority are single customer API integrations (i.e. volume customers who bill against a single account), we are starting to have more interest from third parties who want their customers to integrate their personal accounts with our service. Sound familiar? Of course it does - many great web services on the internet do that already. Facebook, LinkedIn, Flickr, and many more online services have integrations with countless numbers of third party providers. However, customers don’t wish to share their Facebook password with all of these third party sites. How do they let the third party site consume their data on their behalf (term: impersonate)?

OAuth

They use an open protocol called OAuth. OAuth is a clever way of performing a handshake and passing signed messages between servers to obtain an access token to perform activities on behalf of the requestor. Your password is entered only at the site where the password is stored, yet the site requesting access will still be able to access your data in a secure way as if it were you accessing the data.

OAuth is the perfect solution to my authentication problem. There’s just one issue: it takes time to setup and implement correctly. In no particular order, the steps include:

• Setting up database tables to store:
  • Registered OAuth Consumers
  • Consumer Requests
  • Consumer Access Tokens
• Writing a web site to handle user requests
• Writing the backend supporting logic to handle requests and access tokens
• Implementing a workflow for registering OAuth consumers

That’s a tremendous number of tasks. While I would love to fully implement an OAuth provider, I just do not have the time. It would easily take me a couple of weeks at the pace I currently work (about 15 hours per week). So, I have to consider alternative options.

HTTP Basic vs. Sessions

The simplest solution is to send the username and password with each request using HTTP Basic. It’s quick to implement, it doesn’t require any complex behaviors on the client or the server (unlike OAuth), and it’s widely supported by every browser and HTTP client. Here’s what a raw request would look like using Basic authentication:

1
2
3

GET /api/request/12345 HTTP/1.1
Host: api.example.com
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

It’s very simple and light. Now, to the naked eye, you can’t see that the username is Aladdin and the password is open sesame. However, that encoded string is simply Base64 encoded version of Aladdin:open sesame. Essentially, it’s plain text, meaning anyone along the way to the destination can sniff the header and get my username and password. This is a significant concern, but is lessend when you enforce SSL connectivity. However, the risk is still there. Traditional security practices dictate that as the frequency of the transmission of credentials increases, so does the risk of them being compromised, especially if those credentials are in plain text. Thus, from a security perspective, HTTP Basic Authentication is not the best solution. The general security recommendation is if you can minimize the number of times your credential goes across the wire, the risk decreases.

Many posts online give consideration to having a single RESTful resource perform authentication and create a session for the user. Thus, the username and password would be sent down once at the beginning of the session. A cookie is then sent back each subsequent time to identify the user’s session. This is something I absolutely don’t like: RESTful web services are supposed to be state-less. Therefore, having a session active for any period of time violates this principle. Another argument in favor of using this method of authenticating clients is it reduces the request body. Again, in order for a session to be stateful over HTTP, you have to submit a cookie with each request. Cookies are fairly small in size, but when you compare that to a username/password combination in HTTTP Basic sent on every request - it’s about the same size.

HMAC

Amazon’s AWS took a very interesting approach to dealing with security, called Hash-based Message Authentication Code (HMAC). The idea is very similar to how RSA encryption works (e.g. your standard SSL encryption when working with web sites). Amazon’s specific implementation relies on a prior exchange of public and private keys that only the client and the server should know. The idea is the request body plus some additional parameters (such as date, user id, etc.) are hashed using the private key. This payload is sent to the server, where the server generates the same hash from the request body (including those parameters) and checks to see if the hash matches. If it does, it executes the request. Riyad Kalla wrote an excellent blog post explaining this in much greater detail that I suggest you check out.

From a security perspective, this definitely solves the problem with not sending credentials over the wire. It’s simpler than implementing OAuth, as I don’t need additional database tables to support requests and access tokens. Yet, the work necessary on both the client and server-side is much more significant than HTTP Basic authentication. I can implement this solution in a reasonable amount of time. However, I’m significantly concerned about user adoption of the new services. For them to have to learn and implement HMAC to ensure absolutely secure web services will certainly drive-down the number of users who are willing to consume my web services. It’s possible they still will consume my services. Plus, the security-conscious will love this as a solution.

Considerations

All in all, I had several options to weigh. One of the first that had to be considered was how secure my customer’s credentials really needed to be. What types of services am I securing with their credentials? In reality, the type of content being secured is not highly classified materials or supremely private, personally-identifyable information. They are basic requests to start, or check on, a workflow. Future service offerings through this API will allow interactions such as managing your account, getting lists of completed workflows (and associated data), but will still be low-risk information.

In short, I opted to use HTTP Basic authentication. I plan on enforcing SSL connections to the web service. While this is still subject to main-in-the-middle attacks and can result in exposure of a user’s credentials, the risk and gains of doing that are low enough to warrant this as an acceptable solution. Some customers may totally dislike this as an option, and that’s fine. They can continue using the just-as-insecure XMLRPC services until I have a chance to spin up OAuth.

Implementing in Zend Framework

Thankfully, Zend Framework makes adding a provider VERY easy. By creating a class that extends Zend_Controller_Plugin_Abstract, we can hook into the preDispatch pipeline and validate the Authorization header at that point. The code sample shown at the end of this post is fairly self-explainatory. However, I’ll break it down here.

Checking for Authorization Header

We first need to check to make sure the header even exists and is in the proper format. The proper format is Base64 encoded, with the username and password being separated by a colon. That’s what the following bit of code does. Note: the redirectNoAuth and redirectInvalidRequest are helper methods to forward to the appropriate error controller actions.

1
2
3
4
5
6
7
8
9
10
11

$authorizationHeader = $request->getHeader(self::AUTHORIZATION_HEADER);
if ($authorizationHeader == null || $authorizationHeader == '') {
    $this->_redirectNoAuth($request);
    return;
}

$authorizationHeader = base64_decode($authorizationHeader);
if (!preg_match('/[^\:]*\:.*/i', $authorizationHeader)) {
    $this->_redirectInvalidRequest($request);
    return;
}

From there, we extract the username and password from the authorization header and attempt to authenticate. The \My\Auth\Adapter is a \Zend_Auth_Adapter_Abstract, so the logic of authenticating our users is encapsulated there.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$authorizationParts = explode(':', $authorizationHeader);
$username = $authorizationParts[0];
$password = $authorizationParts[1];

try {
    $authAdapter = new \My\Auth\Adapter($this->_customreRepository, $username, $password);
    if ($authAdapter->authenticate() != \Zend_Auth_Result::SUCCESS) {
        $this->_redirectNoAuth($request);
        return;
    }

    $user = $this->_customerRepository->findOneByUsername($username);
    \Zend_Registry::set(self::IDENTITY_KEY, $user);
} catch (\Exception $e) {
    $this->_redirectNoAuth($request);
    return;
}

At the end of that try block, you’ll notice that I use the customer repository to fetch the user by his username and store that in the \Zend_Registry. This is for use later in the APIs when I need to know which user has authenticated with the service, so I can apply authorization constraints against that user. That’s ultimately the only reason you’ll notice the \Doctrine\ORM\EntityManager and the \My\Entity\Repository\CustomerRepository being used in this class. It’s consumed by our authentication adapter and used to find the entity when we’ve authenticated them.

That’s pretty much it. The full class compliment can be found below. I welcome any feedback or suggestsions to this article!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

<?php
namespace My\Controller\Plugin;

use \Doctrine\ORM\EntityManager,
    \My\Entity\Repository\CustomerRepository,
    \My\Auth\Adapter;

/**
 * HTTP Basic Authentication Plugin for My API
 */
class BasicAuthPlugin extends \Zend_Controller_Plugin_Abstract
{
    const AUTHORIZATION_HEADER = 'Authorization';
    const IDENTITY_KEY = 'Identity';

    /** @var \Doctrine\ORM\EntityManager */
    private $_entityManager;

    /** @var \My\Entity\Repository\CustomerRepository */
    private $_customerRepository;

    public function __construct(EntityManager $entityManager, CustomerRepository $customerRepository)
    {
        $this->_entityManager = $entityManager;
        $this->_customerRepository = $customerRepository;
    }

    public function preDispatch(\Zend_Controller_Request_Abstract $request)
    {
        $authorizationHeader = $request->getHeader(self::AUTHORIZATION_HEADER);
        if ($authorizationHeader == null || $authorizationHeader == '') {
            $this->_redirectNoAuth($request);
            return;
        }

        // The header needs to be base64 decoded, then match the regex in order to proceed.
        $authorizationHeader = base64_decode($authorizationHeader);
        if (!preg_match('/[^\:]*\:.*/i', $authorizationHeader)) {
            $this->_redirectInvalidRequest($request);
            return;
        }

        $authorizationParts = explode(':', $authorizationHeader);
        $username = $authorizationParts[0];
        $password = $authorizationParts[1];

        // Authenticate.
        try {
            $authAdapter = new \My\Auth\Adapter($this->_customerRepository, $username, $password);
            if ($authAdapter->authenticate() != \Zend_Auth_Result::SUCCESS) {
                $this->_redirectNoAuth($request);
                return;
            }

            // Get the user and set their identity in the registry.
            $user = $this->_customerRepository->findOneByUsername($username);
            \Zend_Registry::set(self::IDENTITY_KEY, $user);
        } catch (\Exception $e) {
            $this->_redirectNoAuth($request);
            return;
        }
    }

    /**
     * Redirects users to our no authentication error page.
     *
     * @param \Zend_Controller_Request_Abstract $request
     * @return mixed
     */
    protected function _redirectNoAuth(\Zend_Controller_Request_Abstract $request)
    {
        if ($request->getModuleName() == 'api' &&
            $request->getControllerName() == 'error' &&
            $request->getActionName() == 'noauth') {
            return;
        }

        // Forward the request to the noauth error page.
        $request->setModuleName('api')
                ->setControllerName('error')
                ->setActionName('noauth');
    }

    /**
     * Redirects users to our invalid request error page.
     *
     * @param \Zend_Controller_Request_Abstract $request
     * @return mixed
     */
    protected function _redirectInvalidRequest(\Zend_Controller_Request_Abstract $request)
    {
        if ($request->getModuleName() == 'api' &&
            $request->getControllerName() == 'error' &&
            $request->getActionName() == 'invalid') {
            return;
        }

        // Forward the request to the noauth error page.
        $request->setModuleName('api')
                ->setControllerName('error')
                ->setActionName('invalid')
                ->setParam('message', 'Invalid authentication header.');
    }
}