Tech and me

How to do Visual Best Bets for built-in SharePoint Search

2012-01-27T12:13:00.000+01:00

Marcus Johansson did a post about the use of Best Bets this morning which inspired me to write this post. Read his post for more thoughts around the use of best bets.

Visual Best Bets is a feature of FAST Search for SharePoint which lets you point to a file with html content to be displayed above your search results. For example an image, silverlight or flash content can be used to graphically enhance what is linked to the keyword term. The Visual Web Part uses an iframe to accomplish this and loads up your content inside the iframe. This is useful as you can easily edit the html file at will.

But why go the extra mile for a separate file, or opt in for FS4SP for this feature? The Best Bet web part support the showing of keywords and keyword definitions. Keyword definitions are formatted as html. And a definition with html formatting is in effect a Visual Best Bet. (If you have more than one Visual Best Bet you want to assign to the keyword you would have to add them all to the same html for this to work.)

If we examine the Add Keyword page you can format the text for the definition, but there is no button to allow editing the html directly in the style panel.

Will this stop us? No, because we have PowerShell! In a SharePoint management shell get hold of the added keyword “fs4sp” and look at the properties:

PS C:\Users\svc_spadmin> $keyword = Get-SPEnterpriseSearchQueryKeyword -Site http://test "fs4sp"
PS C:\Users\svc_spadmin> $keyword  | Format-List


Term       : fs4sp
Definition :
Contact    :
StartDate  : 26.01.2012 23:00:00
EndDate    : 31.12.9999 23:59:59
ReviewDate : 31.12.9999 23:59:59
BestBets   : {}
Synonyms   : {}
ExpiryDate : 31.12.9999 23:59:59

As you can see there is a property called “Definition”. This parameter holds html, but is empty as I didn’t add a definition via the UI.

For my Visual Best Bet I will add the following html to my $keyword object and update it:

PS C:\Users\svc_spadmin> $keyword.Definition = '<div><img src="http://test/PublishingImages/FAST-search-server
-2010.png" style="float:left"><b>Working with Microsoftr FAST Search Server 2010 for SharePointr, 1st Edition<
/b><i>By Mikael Svenson, Marcus Johansson, Robert Piddocke</i>Deliver the tools clients need to navigate large
 stores of information on your Microsoftr SharePointr implementation-using Microsoft FAST Search Server 2010 f
or SharePoint.<br>Format: Print<br>Publish Date: March 2012</div>'
PS C:\Users\svc_spadmin> $keyword.Update()

Next edit the Search Best Bet web part and make sure “Display definition” is checked and that “Display keyword” is unchecked.

Do a search for “fs4p” and the result looks like this:

If you want to use an iframe pointing to a separate file you can add html like:

<IFRAME title="Search Visual Best Bet" height="150" src="http://fs4spbook.com/" frameBorder="0px" scrolling="no"></IFRAME>

where the src tag points to your file to mimic the behavior of the Visual Best Bets web part.

All part of a days work using configuration!

Returning Best Bets and Visual Best Bets with KeywordQuery and FAST Search for SharePoint

2011-12-13T23:06:00.000+01:00

When using the Search Center the Best Bets or Visual Best Bets functionality works like a charm, but I have not been able to get it to work using the KeywordQuery class in SharePoint.

I had done some research previously on the matter after being contacted by Xavier Vanneste who blogged about this (http://blog.xvanneste.com/Lists/Billets/Post.aspx?ID=82), but was not able to figure it out at the time.

I’m currently writing sample code for my upcoming book on FS4SP and I had to figure this out. Time to bring out the tools!

One hour with the Visual Studio Reflector plugin and setting break-points internally in the SSA I finally figured it out. And in a way it was obvious.

All keyword settings with FAST requires a Search Setting Group and you will have one group for each site collection in your SharePoint farm. The ID of the search setting group is the GUID of the site collection. And this ID is the solution.

KeywordQuery has a property named “UserContextGroupID”, and MSDN states:

Gets or sets the identifier used to group keywords used for matching best bets and visual best bets. This property specifies an identifier used to group keywords used for matching best bets and visual best bets.

By assigning the UserContextGroupID property and requesting to get back SpecialTermResults it is now working like a charm.

Here’s a small sample which will return a Visual Best Bet assigned to the term “FAST Search for SharePoint”.

var searchProxy = (SearchServiceApplicationProxy)SearchServiceApplicationProxy.GetProxy(SPServiceContext.Current);
KeywordQuery kq = new KeywordQuery(searchProxy);
kq.QueryText = "FAST Search for SharePoint";
kq.ResultsProvider = SearchProvider.FASTSearch;

// specify best bets/visual best bets
kq.ResultTypes = ResultType.SpecialTermResults;
// specify the Search Settings Group ID
kq.UserContextGroupID = SPContext.Current.Site.ID.ToString();

ResultTableCollection resultTableCollection = kq.Execute();
ResultTable visualBestBetsTable = resultTableCollection[ResultType.VisualBestBetsResults];
foreach (DataRow row in visualBestBetsTable.Table.Rows)
{
    string name = (string)row["Name"];
    string uri = (string)row["Uri"];
    string description = (string)row["Description"];
    string keyword = (string)row["Keyword"];
    string teaser = (string)row["Teaser"];
    string contentType = (string)row["TeaserContentType"];
}

How to prevent an item from being indexed with FAST for SharePoint

2011-12-10T21:07:00.000+01:00

Yes, it can be done!

It’s Saturday, my kid has gone to sleep, and I finally have time to tell you guys the good news. Preventing an item from being index, or to paraphrase, to drop a document in the document processing pipeline is indeed possible!

You can already prevent items from being indexed by limiting SharePoint lists and libraries from being crawled with library settings and you can use crawl rules to exclude certain url patterns. But what I am talking about is preventing an item from being indexed based business rules in your organization and looking at the meta data of the item or inside the text of a file.

There are many scenarios for not wanting an item searchable. You might want to prevent indexing items in your organization which contains the super secret codename “wobba”, or items of a certain ContentType. When indexing file shares you don't have much meta data to go on at all for excluding items, so creating your own module with the proper rules might be the only way to go.

Up until this post, this was not easily doable with FAST for SharePoint. With built-in SharePoint Search it’s still impossible (unless you create a custom crawler).
There has been at least a couple of threads about this at the FS4SP TechNet forum

and we have long time concluded that this is not an easy task, and cannot be done in a supported manner. (Supported meaning not editing config files which the documentation on TechNet tells us not to touch.)

I had sort of thought about how to do this earlier, but I didn’t figure it out before reading Leonardo Souza’s blog post the other day about: How "Remove Duplicate Results" works in FAST Search for SharePoint.

Leo talks about a property called “documentsignaturecontribution” which can be used to append data to the document signature checksum in the FS4SP content processing pipeline. But in order to assign data to this property you have to create a managed property by that exact name, and output your custom data to a crawled property of your choosing which is mapped to the managed property.

The reason why you have to work with a managed property is because the document signature stage appears after the stage which maps crawled properties to managed properties, and all stages below the mapper stage works on managed properties. This find by Leo is just so cool, and there is no documentation on this anywhere as far as I’ve seen.

So, over to our problem. Which stage runs just before the document signature stage and comes to our aid?

The “Offensive Content Filter” stage

This stage also has an additional attribute where you can assign data, named “ocfcontribution”. There’s only vague documentation on MSDN on how to assign data to this field, which refers to using the XMLMapper. Using the XMLMapper means indexing xml documents, and this is a bit limiting.

The thing about the offensive content filter is that it will prevent documents from being indexed if they contain a certain about of bad language. If you get embarrassed by such words, then skip reading :)

So now we have a stage which can drop items, the rest is to assign enough bad words to “ocfcontribution” to get above the threshold it triggers on.

First off enable the Offensive Content Filter by editing C:\FASTSearch\etc\config_data\DocumentProcessor\optionalprocessing.xml

Next create a managed property called “ocfcontribution” of type “Text”, and also a crawled property with this name. The guid for the property set is one I have chosen for a test group in my system. Replace it with to suit your own system.

$mp = New-FASTSearchMetadataManagedProperty -Name ocfcontribution -Type 1
$cp = New-FASTSearchMetadataCrawledProperty -Name ocfcontribution  -Propset fa585f53-2679-48d9-976d-9ce62e7e
19b7 -VariantType 31
New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $mp -CrawledProperty $cp

In order to test this I have created an xml file named “drop.xml” which I placed in C:\FASTSearch\pipelinemodules with the following contents

<?xml version="1.0" encoding="utf-8"?>
<Document>
   <CrawledProperty propertySet="fa585f53-2679-48d9-976d-9ce62e7e19b7" propertyName="ocfcontribution" varType="31">fuck shit porn cunt cock dick</CrawledProperty>
</Document>

Next I added the following custom extensibility stage to C:\FASTSearch\etc\pipelineextensibility.xml

<Run command="copy C:\FASTSearch\pipelinemodules\drop.xml %(output)s">
    <Output>
        <CrawledProperty propertySet="fa585f53-2679-48d9-976d-9ce62e7e19b7" varType="31" propertyName="ocfcontribution"/>
    </Output>
</Run>

This stage will for each item assign the contents of drop.xml to “ocfcontribution”, effectively dropping all items, which is ok for test purposes. You would of course create a custom module instead which has your business rules for when an item should be dropped.

Next issue “psctrl reset” to reload your config files and use for example “docpush” to index an item, and it will not be indexed, as the output below shows.

PS C:\temp> docpush -c test sample.txt
[2011-12-10 20:31:02.677] ERROR      test Reported error with http://cohowinery.com/sample.txt: processing:OffensiveConte
ntFilter:ERROR: Processor error status: NotPassing
[2011-12-10 20:31:03.678] INFO       test All add operations completed

I hope someone will find this trick useful, and it seems you can use English words for to trigger the filter, no matter the language of your items.

PS! If you enable the Offensive Content Filter and have content with explicit language, you could risk some items not being indexed with this method.

Creating a Secure Store Target application with access for all users

2011-11-28T21:09:00.001+01:00

I am currently working on a SharePoint application which access users Exchange calendars via web services. In order to access the calendars we need to provide credentials and we use a service account which has access to all calendars.

Instead of storing the logon information of the service account in a list or web.config we want to store this in Secure Store which is ideal for storing credentials (big surprise there). We also want to create the Secure Store target application in code with an installer and not using Central Admin.

This is all very straight forward. First we get a reference to the Secure Store with the following code

SPServiceContext context = SPServiceContext.GetContext(SPServiceApplicationProxyGroup.Default, SPSiteSubscriptionIdentifier.Default);
SecureStoreServiceProxy ssp = new SecureStoreServiceProxy();
ISecureStore iss = ssp.GetSecureStore(context);

And in order to create the Target Application we use the CreateApplication method of the ISecureStore object.

void CreateApplication(TargetApplication application, IList<TargetApplicationField> applicationFields, TargetApplicationClaims applicationClaims);

The issue comes when you want to create the claims going in as the third parameter. In our case we want all users to use the same credentials in the form of all authenticated users and I’ve create a “group” type target application. For “group” type target applications we have to set the administrator and member claims on the TargetApplicationClaims object.

But what should we pass in? If you use the wizard in SharePoint admin to set the credentials, the user picker control will do the conversion for us from whatever user or group we enter. When programming this ourselves it’s a different matter.

My trusty old friend Reflector helped me out and I found the perfect claims provider for my use.

Microsoft.SharePoint.Administration.Claims.SPAllUserClaimProvider

This class has a static method named “CreateAuthenticatedUserClaim” which return a claim for all authenticated users. (Try Google the method name and see how much help you get).

The final code looks something like this:

TargetApplication app = new TargetApplication("appid", "friendlyname", "admin@email.com", 0, TargetApplicationType.Group, null);
List<TargetApplicationField> fields = <the fields in the target app>;
string administator = "DOMAIN\\adminuser";
var adminWindowsClaim = SPActiveDirectoryClaimProvider.CreateUserClaim(administator);
SecureStoreServiceClaim adminSecureClaim = new SecureStoreServiceClaim(adminWindowsClaim);

SPClaim groupWindowsClaim = SPAllUserClaimProvider.CreateAuthenticatedUserClaim(true);
SecureStoreServiceClaim groupSecureClaim = new SecureStoreServiceClaim(groupWindowsClaim);

TargetApplicationClaims apc = new TargetApplicationClaims(new List<SecureStoreServiceClaim> { adminSecureClaim }, new List<SecureStoreServiceClaim> { groupSecureClaim }, new List<SecureStoreServiceClaim>());
iss.CreateApplication(app, fields, apc);

Taking promotions/demotions one step further in FAST Search for SharePoint

2011-11-20T20:51:00.000+01:00

I am not sure how many times I have read the documentation on promotions and demotions in Microsoft documentation, but I can say it is numerous.

And yet yesterday I discovered a valuable piece of nugget which I’m sure I will use a lot in the future.

Up until yesterday this was my complete understanding of promotions and demotions in FAST for SharePoint.

1.
You can promote and demote items based on it’s url. Via SharePoint admin UI the boost is absolute by adding 1,000,000 rank points, effectively pushing the item to top of the result list.
If using the SharePoint UI you have to add the promotions to a keyword, but when using PowerShell you can make a “global” promotion which affects all search queries.

2.
You can promote and demote sites based on the start of a url. For example promoting “http://site/” via SharePoint admin UI will give all items which has a url starting with “http://site/” 1,000 extra rank points. A soft boost.

3.
Using PowerShell to create promotions and demotions lets you specify the boost value yourself.

4.
Each of the promotions can be associated with a user context in order to target the promotion to specific users, and you can set a start and end date for the promotion.

There is MORE!

This is all good, but there is more. Doing some PowerShell scripts around promotions I stumbled upon the very little documented function AddPromotedExpression. If you google this function you will get two hits, both on MSDN with the very sparse documentation text

“Creates a promoted expression and adds it to the collection.”

with the function signature:

PromotedExpression AddPromotedExpression(
    string fqlExpression
)

What this means is we can write an arbitrary FQL expression for promoting or demoting items. Let’s say we want to demote all items which are larger than 3MB and located on a file server. The following fql will match the criteria:

and(size:range(int("3145728"),max,from="gt",to="le"),url:starts-with("file://"))

Executing the following lines in a FS4SP PowerShell will add the demotion:

$searchSettingGroup = Get-FASTSearchSearchSettingGroup
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
$globalPromotion = $globalPromotions.AddPromotion("FQL size and file") 
$globalPromotion.BoostValue = "-4000"
$fql = 'and(size:range(int("3145728"),max,from="gt",to="le"),url:starts-with("file://"))'
$globalPromotion.PromotedItems.AddPromotedExpression($fql)

Hope this will come in handy!

Error while running the FAST for SharePoint post configuration

2011-10-21T22:09:00.002+02:00

During a customer installation the other day my project colleagues encountered a weird issue when configuring up a FAST server. (This was after we got all policy issues resolved due to server hardening – which took some time as well).

When running the configuration wizard they got an unexpected error, and the configuration didn’t finish.
Time to hit the installer log…. The error they got was hidden a bit down the load and was coming from the “ULS Common Core Components” which is installed during configuration.

20.10.2011 14:42:39 Verbose InstallAction - Installing ULSCommonCore services
20.10.2011 14:42:39 Verbose Executing ULSCommonCore installer msiexec /i "C:\FASTSearch\\installer\files\ulscommoncore.msi" /quiet /norestart INSTALLLEVEL=100 - 
20.10.2011 14:42:39 Verbose Utility.Execute - Starting process msiexec with working directory - , write output - False, file - , user - 
20.10.2011 14:42:41 Warning Utility.Execute - Return code for binary msiexec is not 0. This may indicate that binary didn't execute successfully
20.10.2011 14:42:41 Verbose Utility.Execute - Finished executing msiexec
20.10.2011 14:42:41 Error InstallULSCommonCore - An error occurred while executing binary msiexec. Return code is not 0.
20.10.2011 14:42:41 Error Utility.WriteException - Exception -  : Exception - System.Management.Automation.RuntimeException: Error executing ULSCommonCore installer.

Upon my suggestion they kicked off the ULS installer manually and got the following pop-up which gave information for further internet searches.

The error is caused by the registry size being too small and has to be increased. Not too obvious from neither the install log nor the pop-up.

Thanks to the below post for giving the resolution on how to increase it :) http://www.geosoft.com/support/knowledge-base/errormessage/Error-An-error-occurred-during-the-installation-of-assembly-Microsoft-VC90-ATL.

A side-by-side approach for upgrading SharePoint search to FAST

2011-10-17T21:15:00.001+02:00

More and more businesses are upgrading their existing SharePoint 2010 search solutions to FAST Search for SharePoint. Doing so will most likely require scheduled downtime when you make the actual switch.

This can be avoided by cleverly upgrading your existing Search Service Application to a FAST Query SSA.

Want to learn more? Head over to my article at NothingButSharePoint.com for the details - http://pzl.no/qKacHz.

NSC Talk–SharePoint search planning 101

2011-09-21T22:11:00.001+02:00

Today was an excellent meeting at the Norwegian SharePoint Community where I had the pleasure to do one of three talks for the evening.

The first talk was by Henning Strand from Mills. Henning talked about their process portal to support product development, a portal made with very little code. The second talk was by my colleague Mads Nissen, outlining how we at Puzzlepart do apps development for SharePoint without using SharePoint – only ASP.Net and Excel. Both talks were very interesting and targeted both end-users and devs.

Last one out was myself doing a slide only talk on SharePoint search planning, where I tried to level the crowd from level 100 to level 400 on the SharePoint maturity model regarding search, in just 25 minutes.

Henning and Mads both went over their allotted time so I did my presentation in a lightning fast tempo. My Cliffs Notes reference went above everyone's heads as they had never heard about it, but I hope the crowd got the message in spite of me talking extremely fast. I talk quite fast when getting excited, and trying to catch up the lost time didn’t help :)

Also thanks to Michal Pisarek for letting me "steal" from “Why Your SharePoint Search Sucks”, a slide deck worth going over. Mine was sort of the Cliffs Notes of his talk. Both talks about the same topic but with different approaches.

How would you optimize this piece of C# code?

2011-09-06T20:50:00.002+02:00

I’ve been digging into some code lately when getting into an existing project, and came across these two lines:

if (Math.Max(thisString.Length, otherString.Length) > Math.Pow(2, 31))
throw new ArgumentException("String too long");

Looking at the code I see right away that this is ported from some other language over to C#. It’s quite easy to optimize it…. how would you do it?

Adding the left navigation menu to web part pages in SharePoint 2010

2011-08-30T12:39:00.000+02:00

I taught a SharePoint 2010 super-user class last week, and the students experienced like so many before them that the web part page layouts which come with SharePoint 2010 removes the left hand navigation menu (the quick launch).

This is nothing new and it has been blogged about since 2007. But on my 15 minute Google spree this morning I could not find a recipe which worked 100% with SharePoint 2010, so I will give it a try myself, and hopefully this can help some others in the future. I’m sure someone else has a complete post on this which I missed, but at least I have leveled up myself in the use of SharePoint Designer and modifying existing layouts instead of creating custom ones :)

Step 1 – Create a new web part page

Create a new web part page either via SharePoint or SharePoint Designer

Step 2 – Edit the page in SharePoint Designer

Edit your web part page in “Advanced Mode” in SharePoint Designer

Step 3 – Remove custom css

Around line 34 you will find a code block like the one below

<SharePoint:UIVersionedContent ID="WebPartPageHideQLStyles" UIVersion="4" runat="server">
<ContentTemplate>
<style type="text/css">
body #s4-leftpanel {
display:none;
}
.s4-ca {
margin-left:0px;
}
</style>
</ContentTemplate>
</SharePoint:UIVersionedContent>

Delete this code block.

Step 4 – Remove the overrides for the left column

Further down you will find three lines which prevents the left column for rendering.

<asp:Content ContentPlaceHolderId="PlaceHolderPageImage" runat="server"></asp:Content>
<asp:Content ContentPlaceHolderId="PlaceHolderNavSpacer" runat="server"></asp:Content>
<asp:Content ContentPlaceHolderId="PlaceHolderLeftNavBar" runat="server"></asp:Content>

Remove all three lines.

Step 5 –Save the page

Save and publish your page.

Step 6 – Preview your changes

Why you should use deep refiners with FAST for SharePoint

2011-08-11T09:25:00.001+02:00

With FAST for SharePoint you have what is called deep refiners, compared to shallow refiners which the built-in search uses. The difference is that deep refiners guarantee an exact refiner count, and all possible refiner values for your search, while shallow will calculate the counts based on the first 50 hits (default setting which can be adjusted). In addition to deep refiners, FAST for SharePoint also supports shallow refiners.

The first image shows a query using FAST for SharePoint with deep refiners and the second image show the built-in SharePoint Search with shallow refiners.

With FAST for SharePoint deep refiners are stored in a separate data structure (mostly in memory), optimized for just that; returning the refiners. Think of it as a pre-calculated refiner tree, ready for the picking.

Shallow refiners on the other hand have to be manually constructed by the query server from the results returned. Returning a large result set, looking at the metadata and building the refiners, is both IO and CPU intensive, as it has to retrieve all metadata per item, and then build the refinement result based on that.

Of course, basing the refiners on 50 results is a speedy operation, but at the cost of the accuracy of deep refiners.

So, when you create a refiner with FAST for SharePoint, be sure to tick off the “Deep refiner” checkbox to take advantage of the refiner structure stored within the search index. You will not only get more accurate results, but in most cases they will also return even faster than shallow ones.

PS! There are cases when deep refiners will be slow, particularly if you have very many unique values within a refiner. Then again, if this is the case, you might want to re-think your search solution.

(Reference: http://technet.microsoft.com/en-us/library/gg193929.aspx)

Working with Content Collections via the API–Continued (and yet another bug)

2011-08-10T11:28:00.006+02:00

Yesterday I wrote about how you could use the admin API to create a new content collection for FAST Search for SharePoint. I explored this further, and discovered a more serious bug. From within a SharePoint context it will never work (see the end of this post for the solution)

For the code sample yesterday I instantiated the ContentContext with the line

ContentContext contentContext = new ContentContext();

Using the default constructor like this implies that the code is run on a server which has FAST Search for SharePoint installed as it will read the configuration file from %FASTSEARCH%\etc\Admin.config.

When using the ContentContext object from within SharePoint you have to go via the FASTAdminProxy property on the SearchServiceApplicationProxy object from the FAST Query SSA. If this sounds greek to you, the code below might shed some light.

var ssaProxy = (SearchServiceApplicationProxy)SearchServiceApplicationProxy.GetProxy(SPServiceContext.Current);
if (ssaProxy.FASTAdminProxy != null)
{
   var fastProxy = ssaProxy.FASTAdminProxy;
   // Get a reference to the ContentContext via a WCF proxy
   ContentContext coll = fastProxy.ContentCollectionContext;
   coll.Collections.AddCollection("notes", "Lotus notes connection");
}

The FASTAdminProxy reads it’s WCF configuration from C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebClients\FASTSearchAdmin\ContentCollectionService\client.config.

But, now we encounter yet another bug. The factory class which creates the proxy has a readonly member “ServicePostfix” which returns the name of the WCF service endpoint. For the ContentContext it returns “ContentCollection.svc”.

Microsoft.Office.Server.Search.Administration.FASTAdminProxy

ContentCollectionServiceFactory
      ServicePostfix

But on the FAST server the endpoint is named “ContentCollectionService.svc”, so we get an error that it cannot connect to the WCF service. The fix is simple and involves copying the file

%FASTSEARCH%\components\admin-services\contentcollectionservice.svc

over to

%FASTSEARCH%\components\admin-services\contentcollection.svc

How-to: Create a content collection in code

2011-08-09T15:41:00.001+02:00

(Update: Read http://techmikael.blogspot.com/2011/08/working-with-content-collections-via.html for how to do this in a SharePoint context)

When installing FAST Search for SharePoint it creates a default content collection named “sp” where all content crawled via the FAST Content SSA is stored.

You also have the option to create new collections, and this is typically something you would do for the FAST Specific connectors (FAST Enterprise Web Crawler, FAST Database Connector, FAST Lotus Notes Connector) in order to support management like clearing out all content.

A content collection is merely a logical grouping of content inside of FAST Search for SharePoint, where all items indexed have an additional field named “meta.collection” attached, and not something which affects the physical layout of how FAST stores the search index.

In PowerShell you can use the command New-FASTSearchContentCollection to create new collections, but sometimes you want to do this in code for example via SharePoint feature activation.

Once you have a reference to the ContentContext object, it’s one line of code to create a new collection:

ContentContext contentContext = new ContentContext();
contentContext.Collections.AddCollection("notes", "Lotus notes connection");

Except if you run this code you get an exception:

Unhandled Exception: Microsoft.SharePoint.Search.Extended.Administration.Common.AdminException: Invalid pipeline name 'Office14'.

The name “Office14” is a static string retrieved from Microsoft.SharePoint.Search.Extended.Administration.Content.CollectionConstants.Office14

So how do we fix this? Examining the pipelines installed for example using PowerShell, we find the correct name of the pipeline:

PS C:\FASTSearch> Get-FASTSearchDocumentProcessingPipeline
Name
----
Office14 (webcluster)
Attachments (webcluster)

The correct name is “Office14 (webcluster)”, not “Office14”. Modifying the initial code to use the overload which also takes in the name of the pipeline to use solves the error:

contentContext.Collections.AddCollection("notes", "Lotus notes connection", "Office14 (webcluster)");

The “webcluster” part of the name is something which i s carried over from FAST ESP.

Updated sample deployment files with FAST Search for SharePoint SP1

2011-08-08T09:25:00.001+02:00

With service pack 1 (or the June 2011 CU) the sample deployment files included with FAST for SharePoint has been updated. The single server sample is still the same, as there is not much room for changing where you put components, but both the multi-server samples have some changes.
The changes align better with recommend best practice according to the number of nodes used in the samples. Depending on your requirements you can use the provided samples as starting points and modifying them to your needs.

deployment.sample.multi1.xml

This is a sample configuration for a two server deployment with search redundancy.

Removed failover on indexing

The second host has removed <content-distributor> and <indexing-dispatcher> nodes. In a two node system it is better to have a performant system for searching, letting one of the nodes be a pure search server. The configuration still has failover for searching with two search rows.

The number of document processors is increased from 4 to 8

The recommended setup is one document processor per CPU core, and many servers today come with 8 cores (single quad-core cpu with hyper-threading = 8 visible cores in the OS).

The Enterprise Web Crawler has been removed from the configuration

For most scenarios you will use the web crawler in SharePoint instead of the FAST Enterprise Web Crawler, so there is no need to include it in the deployment configuration.

Max-targets for the webanalyzer has been increased from 2 to 4

Max targets specify the number of CPU's the web analyzer utilizes, and aligns better with the number of cpu’s on a typical server.

Removed failover webanalyzer component

By removing the attribute redundant-lookup="true" from the <webanalyzer> node on the first host, and removing the <webanalyzer> node entirely from the second host, failover of the webanalyzer is removed from the sample configuration. In a two server setup failover on this component is not necessary. If the first host which is responsible for indexing dies, indexing will stop working, and the webanalyzer is dependent on indexing in order to get data to work with.

deployment.sample.multi2.xml

This is a sample configuration for a five server deployment with a separate admin server, search redundancy, feeding redundancy and indexing redundancy. The updated sample has removed redundancy on the web analyzer component.

If a server performing database lookup fails for link analysis, and there is no redundancy, this will block crawling.

Admin server changes

The admin server used to have the Enterprise Web Crawler, but that is now replaced with the web analyzer and 8 document processors.

First indexer node

The web analyzer is moved over to the admin node, and the number of document processors has been increased from 4 to 8.

Second indexer node

Number of document processors increased from 4 to 8, and the failover web analyzer component is removed from the server.

Search nodes

The search node configurations are left as is, only serving search requests.

Creating refinement query parameter for FS4SP by code

2011-08-05T09:48:00.006+02:00

[Update: 2011-12-09]
Someone from somewhere took the time to use Reflector and pull out the appropriate code from the FS4SP dlls. Download the code below.

When using the Search Center with FAST Search for SharePoint you will see the query parameters being modified when you interact with the page.

The k= parameter holds your search query. This is useful if you create a link somewhere on your site which redirects you to the search page with a pre-set query. An example is creating a link which displays all the documents from a particular employee containing the word contract.

This can be accomplished by the query:

contract author:”Mikael Svenson”

which in the k= parameter looks like:

k=contract%20author%3A%22Mikael%20Svenson%22

And this works just fine. What doesn’t work is if you have specified a synonym for the word contract. When tacking on the author parameter to the query, it will no longer match in the synonym list, as the synonyms match the whole query term, not just a single word.

Another approach would be to add the author filter as a refiner instead in the r= query parameter, leaving only the word in the k= parameter which will match the synonym definition.

However, if you look at the r= parameter for a query refining on me as an author it looks like this:

r=author%3D%22AQ5NaWthZWwgU3ZlbnNvbgZhdXRob3IBAl4iAiIk%22

The quick ones out there spot this as being a base64 encoded string. If you decode the string and with some investigation you can find out how it’s being built up.

The structure looks like this:
(char)1
(char)length of property value
property value
(char)length of property name
property name
(char)1
(char)2 //length of the next two bytes
^”
(char)2 //length of the next two bytes
“$

For your convenience I have created a small class which does the heavy lifting for you.

The usage is like this:

BuildRefinerParameter builder = new BuildRefinerParameter();
builder.AddRefiner("format", "Adobe PDF");
builder.AddRefiner("companies", "Microsoft");
string url = builder.GetRefinerQueryParameter();

And the class itself:

class BuildRefinerParameter
{
    Dictionary<string,string> _refiners = new Dictionary<string, string>();
    public void AddRefiner(string key, string value)
    {
        _refiners[key] = value;
    }
    public string GetRefinerQueryParameter()
    {
        StringBuilder sb = new StringBuilder();
        foreach (var refiner in _refiners)
        {
            sb.Append(refiner.Key);
            sb.Append('=');
            sb.Append('"');
            string base64encodedRefiner = CodeRefinerParameter(refiner);
            sb.Append(Convert.ToBase64String(Encoding.UTF8.GetBytes(base64encodedRefiner)));
            sb.Append('"');
            sb.Append(' ');
        }
        string url = HttpUtility.UrlEncode(sb.ToString().Trim());
        return url;
    }
    private static string CodeRefinerParameter(KeyValuePair<string, string> refiner)
    {
        var refkey = refiner.Key;
        var refval = refiner.Value;
        StringBuilder coded = new StringBuilder();
        coded.Append((char) 1);
        coded.Append((char) refval.Length);
        coded.Append(refval);
        coded.Append((char) refkey.Length);
        coded.Append(refkey);
        coded.Append((char) 1);
        coded.Append((char) 2);
        coded.Append('^');
        coded.Append('"');
        coded.Append((char) 2);
        coded.Append('"');
        coded.Append('$');
        return coded.ToString();
    }
}
}

Creating a scope to limit by content collection

2011-07-24T14:23:00.004+02:00

When working with FAST Search Server for SharePoint and especially if you are using one of the custom FAST connectors you will usually create additional content collections with the command: New-FASTSearchContentCollection
This will logically separate content from for example the FAST Enterprise Web Crawler from content indexed from SharePoint into the sp collection (which is the default collection for the FAST Content SSA).

What if you want to show results only from the web crawl? If it had been a content source on the FAST Content SSA you could have created a search scope to limit by content source like this:

New-SPEnterpriseSearchQueryScope -SearchApplication "FAST Query SSA" -Name MyContentScope -Description "Content Source Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter 'contentsource:MyContentSourceName'

Since the FAST specific connectors don’t register as content sources on the FAST Content SSA we have to use an old FAST ESP trick instead. A content collection is merely a logical grouping of content inside the search index, and each collection will have meta data per item informing us of which content collection the item belongs to.

By creating a scope on the meta.collection property, we achieve what we set out to, limit the results per content collection.

If your content collection for the web crawl is named web, then the following PowerShell command would create a scope which filters on the web collection.

New-SPEnterpriseSearchQueryScope -SearchApplication "FAST Query SSA" -Name MyCollectionScope -Description "Collection Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter 'meta.collection:web'

Released: SharePoint Search Parts

2011-07-19T22:52:00.001+02:00

The past couple of weeks I’ve been working evenings and some during the day when my kid is asleep on a FQL (FAST Query Language) enabled Core Results web part. The reason for going fql is not to write it directly into the search box, but in order to provide proper synonym and wildcard support. (Note: fql is only supported with FAST for SharePoint)

The project can be found at http://spsearchparts.codeplex.com/ and is currently in alpha. It might actually be higher quality than what you expect from an alpha version, but I haven’t had time to test it properly.

The biggest undertaking in the project was to write a kql->fql parser. I think the parser works ok at the moment, but there is always room for improvement, and I might have missed something.

The other technical challenge was to enable fql support in the web part. Thankfully, Ivan Neganov has a blog post on how to do this which I used as foundation for the web part. I did however change the use of direct reflection to use IL.Emit, as it’s more performant and can be cached, but it’s still reflection ;-)

Synonyms are expanded based on single words or phrases. If you add a synonym in SharePoint with two or more words, it will only match in “search phrases” with quotes. I might change this to match multiple terms as well. Synonyms are also lemmatized/stemmed, which they are not in the standard web part.

There is also an option to boost the original query words over the synonyms which could be useful. At least it was fun to implement :-)

Wildcards are supported like they are in fql, where both ? and * can be used, and also anywhere in the words.

When using the web part you can use my query tool at http://fs4splogger.codeplex.com/ to inspect how the kql query is transformed into fql. This will also show synonym expansion and added scopes.

I plan to add more search parts as I have time, and hopefully the project can grow into something really useful.

More information can be found at the project page.

Improved relevancy with SP1

2011-07-12T21:30:00.001+02:00

This was listed as one of the updated in SP1, and was actually released in the December CU. So what has really changed?

For all the default Metadata Categories in FAST for SharePoint the default behavior is to map auto-discovered crawled properties to the full-text index.

This leads to many irrelevant fields being searchable. The update is in fact adding many of these fields to an ignore list, setting the IsMappedToContents property of the crawled property to false.

The full list of the added ignored properties can be found in the function ImproveDefaultSchema in C:\FASTSearch\installer\scripts\include\commontasks.ps1.

The relevancy improvement is to avoid garbage into the index.

Thanks to Microsoft for providing this information.

SharePoint MVP for 2011

2011-07-02T20:48:00.001+02:00

Yesterday was the first day of my July vacation between jobs, and I was spending the evening having a good time with some friends of mine.

I arrived at the restaurant 20 minutes early, ordered a beer and checked my e-mail. And there it was, an e-mail from the Microsoft MVP team reading:

Congratulations 2011 Microsoft MVP

This is my first time being awarded, and I have been awarded in the SharePoint Server category. For those who follow my blog you know that FAST for SharePoint is what I do these days, and it’s a community I very much enjoy being a part of.

Getting an award is something to live up to in my opinion, and I will try my best to continue to do what I love

PS! We all had a blast last night!

FAST for SharePoint Service Pack 1

2011-06-29T09:48:00.003+02:00

Together with SharePoint 2010 Service Pack 1, FAST for SharePoint got a service pack on it’s own. you can download it from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=26612
The most notable fixes for me are:

Improves default schema that improves relevancy
Better title and date extraction for Microsoft Office documents
Adds more flexible custom property extractors
Updated sample deployment templates <- Note: My example deployment templates were messed up during the install of SP1
Managed properties does no longer have to be in lowercase in the refinement panel configuration, in the query API’s or in FQL scope filters
Improves index backup and index restore
Improved indexing speed for several scenarios
Fixed index corruption when the indexer crashes
~~Functionality to add search columns to a live system~~ [Edit: Was not included afterall, but will in a later CU]
Infotool.exe will pick up more log files and configuration files

As for the updated custom property extractors, the optionalprocessing.xml configuration file is now obsolete (but backwards compatible) and has been replaced by a new file called CustomPropertyExtractors.xml. (Both reside in C:\FASTSearch\etc\config_data\DocumentProcessor).
In this new file you can add as many extractor dictionaries as you want, not just three whole-word and two word-part dictionaries as was the limit before. And you can also have both case and case-insensitive matching.
There is a migration guide at: Migrate custom extractor to sp1

Error in default important level weights for the Full-text index mappings

2011-06-24T12:36:00.004+02:00

The title for this post might seem somewhat cryptic if you haven’t worked with managed properties in FAST for SharePoint, and mapped them to different priority levels. But I will walk you thru it getting to the error.

First of all, the relevance score for a search hit in FAST for SharePoint is build up of many different values, where one part of the score is how important is the field which contains data where your query matched. As an example if your query matches words in the title it will get rated higher compared to if it matched in the body text of the document.

When you create a managed property which holds textual content, you can set how important this field is, from level 1-7 as seen on the image below. You can get to this screen by going to

Central Admin –> FAST Query SSA –> FAST Search Administration –> Managed properties –> <click a property> –> <scroll to the bottom of the page>

Using PowerShell we can list the weights behind the difference important levels.

$rankprofile = Get-FASTSearchMetadataRankProfile default
$content = $rankprofile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}
$content.GetImportanceLevelWeight(1)
30
$content.GetImportanceLevelWeight(2)
10
$content.GetImportanceLevelWeight(3)
20
$content.GetImportanceLevelWeight(4)
30
$content.GetImportanceLevelWeight(5)
40
$content.GetImportanceLevelWeight(6)
50
$content.GetImportanceLevelWeight(7)
60

As you can see Level 1 has a weight of 30, the same as Level 4, and this is where the error is.

To rule out any magic going on behind the scenes I conducted a test. First I created three crawled properties, which each was mapped to three managed properties, as circled in red in the first image. Then I created three documents with the same content, where levelone.txt was indexed into the madcowone field, leveltwo.txt into madcowtwo and levelfive.txt into madcowfive. I also set the freshness weight to zero, to rule out the time factor on ranking.

When I executed a search against these three documents, they all got the same rank score of 39, but I could see via my FS4SP Query Logger tool that they did get different context scores, but they were sorted on random as you can see in the output (shortened for clarity):

I have highlighted the context score and also the level in which we got a hit, which corresponds to the name of the document. The reason for the low score is that the Context Weight doesn’t count as much compared to other factors in the static rank.

Next I changed the context weight from the default of 50 to 200. Executing the same query I now got these results, sorted in the “correct” order, levelfive.txt, levelone.txt and leveltwo.txt.

This means that the Level 1 field clearly ranks above Level 2 and 3, and this is most likely an error with the product. And you probably want to change the values for the importance levels in your deployments to match the expected behavior.

Deleting a Federated Location via the API

2011-06-22T14:51:00.002+02:00

I’m working on a SharePoint feature which will automatically add a federated search location on feature activation and which will delete it on de-activation.

To accomplish this I use the Microsoft.Office.Server.Search.Administration.SearchServiceApplication class, and more specifically the AddNewLocationConfiguration and DeleteLocationConfiguration methods. I tried to use the same methods of the SearchServiceApplicationProxy first, but there seems to be an error using the AddNewLocationConfiguration method on proxy object.

AddNewLocationConfiguration takes a LocationConfiguration object as parameter, which can be created by loading in the xml from an OSDX file like this:

var searchLocation = new LocationConfiguration();
var stream = …stream to xml file…; searchLocation.Import(stream);

As you see, adding is not too hard, but deleting is the problem. The DeleteLocationConfiguration method takes the id of a LocationConfiguration object as the parameter. But lo and behold, the Id property is defined as internal.

That means it’s not available for us to delete on. But we’re lucky to use .Net where reflection is possible. Buy using the helper class below we can get the value from the internal property. Note that the code could have used regular reflection, but emitting IL makes more optimized code, and I use this utility class a lot of other places where performance is an issue.

public class DynamicReflectionHelperforObject<V>

{
public delegate T GetPropertyFieldDelegate<T>(V obj);

public static GetPropertyFieldDelegate<C> GetP<C>(string memberName)
{
Type v = typeof(V);
PropertyInfo pi = v.GetProperty(memberName, BindingFlags.NonPublic | BindingFlags.Instance);
if (pi == null)
throw new NullReferenceException("No Property or Field");

DynamicMethod dm = new DynamicMethod("GetPropertyorField_" + memberName, typeof(C), new Type[] { v }, v.Module);
ILGenerator il = dm.GetILGenerator();

il.Emit(OpCodes.Ldarg_0); // loaded c, c is the return value
il.EmitCall(OpCodes.Call, pi.GetGetMethod(true), null);
il.Emit(OpCodes.Ret);
return (GetPropertyFieldDelegate<C>)dm.CreateDelegate(typeof(GetPropertyFieldDelegate<C>));
}
}

And to use it pass inn the object type, the property type and the object we want to retrieve the property from:

int id = Helpers.DynamicReflectionHelperforObject<LocationConfiguration>
.GetProperty<int>("Id")
.Invoke(frontLocation);
fastQuerySSA.DeleteLocationConfiguration(id);

Making Best Bets available right away

2011-06-17T13:39:00.002+02:00

In SharePoint 2010 if you add a keyword and attach a Best Bet to it, it is usually made available right away. But if your system has a lot to do, then the timer jobs kicking this into production might not run as often as you desire.

This is easily fixed by running the “Prepare query suggestions” timer job. Seems this jobs handles anything to do with synonyms, nicknames and also keywords.

In a SharePoint Management PowerShell window you can start the timer job by issuing

Start-SPTimerJob -Identity "Prepare query suggestions"

Clearing items from a specific content source

2011-06-15T23:42:00.002+02:00

On your Search Service Application, that be the standard SharePoint Search or FAST for SharePoint has a link called Index Reset which will clear all the searchable items in your index.

For FAST Search for SharePoint you would have to call Clear-FASTSearchContentCollection on the FS4SP farm as well via PowerShell.

This is all fine if you want to remove everything, but sometimes you may want to only remove parts of the index, say only items from a particular content source like SharePoint or your file server.

The trick to achieve this is to remove the start addresses from a search location, and then re-add them. When you remove a start address from a content source, it triggers a delete mechanism which will remove all items from that particular start address.

Here’s a sample PowerShell script which will remove all items from my File server content source called File residing on my Search Service Application called FASTContent. Probably something to turn into a .ps1 file or a cmdlet.

$sourceName = "File"

$contentSSA = "FASTContent"

$source = Get-SPEnterpriseSearchCrawlContentSource -Identity $sourceName -SearchApplication $contentSSA

$startaddresses = $source.StartAddresses | ForEach-Object { $_.OriginalString }

$source.StartAddresses.Clear()

ForEach ($address in $startaddresses ){ $source.StartAddresses.Add($address) }

DateTime resolution on your SharePoint Search Center hits

2011-06-02T21:17:00.003+02:00

If you use the search center in SharePoint you will notice document dates are listed with year, month and day. What if you want to show hours, minutes and seconds as well for the last modified date?

Turns out this is harder than you would imagine as the underlying xml is missing the data we need, so changing the xslt will not help.

If I use the FAST Search for SharePoint Query Logger on my FAST server, I see that hours, minutes and seconds are returned, so the question is, where do they disappear?

Firing up one of my favorite tools, Reflector, I found the answer in the SharePoint result XML builder, an internal class used to build search result xml - Microsoft.Office.Server.Search.Query.Gateway.XMLBuilder. In the method GetPropertyValue there is a section handling the managed property called write.

and if we take a closer look at GetDateTimeValue we find the answer.

In the last line it formats the date with the d modifier. This stands for short date, and will render a date without hours, seconds and minutes -http://msdn.microsoft.com/en-us/library/az4se3k1.aspx. The siteCulture parameter comes from the search runtime (either the SharePointSearchRuntime or the FASTSearchRuntime, depending on if you are using the built-in search or FAST Search for SharePoint).

My first instinct was to override the formatting of the d modifier for the culture of the running thread. But this will not help as it’s the SSA runtime which executes the query and it is running on a different thread.

Seems your only solution when using the Core Results Web Part is to create a new managed property, and map to it the same crawled properties as is mapped to write. Then use your new managed property instead in the xslt rendering.

If you are using any of the API’s for search, you will get the full date resolution, as the result is not run thru the result XML Builder.