Christopher Even on SharePoint Search

Great overview of what's coming for search in SharePoint 2010.

notorioustech — Thu, 12 Nov 2009 14:39:00 GMT

http://blogs.msdn.com/enterprisesearch/archive/2009/10/28/fast-meets-sharepoint-what-s-coming-in-search-for-sharepoint-2010.aspx

Vegas SharePoint 2010 Conference is going to be great!

notorioustech — Sat, 17 Oct 2009 21:57:00 GMT

I hope you all are going to be there to see the exciting new SharePoint 2010 product announcements and demos.

I will personally be giving connector demos in the BA-Insight/3008 booth so stop by if you want to see how to connect your Enterprise Systems to SharePoint and FAST.

We are also demoing our latest connector updates which allow you to push content securely into SharePoint Lists and Document Libraries for content migration and synchronization of any systems with FULL Security translation to SharePoint.

These as are the connectors i will be demoing live for both SharePoint and FAST:

ERP, CRM and Custom Systems
SAP
Salesforce.com/Force.com
Microsoft Dynamics
Interaction
Siebel
Oracle / SQL Server Databases
Web Services

ECM
Interwoven Worksite
EMC Documentum
OpenText Hummingbird
Symantec e-Vault

Messaging Systems
Lotus Notes
Exchange Private Mailboxes

Come see BA-Insight's 2010 integration also with full previews and more.

Connecting FAST and SharePoint Search to Enterprise Systems Presentation

notorioustech — Tue, 29 Sep 2009 14:21:00 GMT

I gave an online presention for BA-Insight yesterday to a global audience and it went very well. The questions at the end and during asked by the attendees showed a great interest on how FAST works with other sytems. Here is the agenda and download.

What is Enterprise Search?
FAST vs. SharePoint?
Seven different ways to connect in your data for FAST or SharePoint Search
Search Security Concepts (Index Level vs. Real Time)
A detailed view on Identity mapping for security (time permitting).
Leveraging data relationships between systems in search.
Impact of cloud based systems on Enterprise Search

Download the presentation powerpoint here, with some notes in the margins.

The presentation was recorded and is available now if you want to what. There was ALOT of information not in the slide deck so check it out:

https://www2.gotomeeting.com/register/136673515

Beware: Adobe 9 ifilter can use up CPUs even when not filtering

notorioustech — Wed, 16 Sep 2009 12:49:00 GMT

I have had this reported from more than a few customers than even when their indexing is complete the mssdmn.exe process which is where the iflters process your files will continue to use up CPUs. I suspect this is a poor attempt by Adobe to keep their libraries and resources loaded by using a timer or sleep loop for faster startup, but it obviously is not working well.

Note: this behaviour was NOT reported for the Adobe 8 reader ifilter.

Solution: Install the Foxit or PDFLIB.com ifilter instead, it will cost you a little money (usually less than 1k) but your problem will be solved.

Cheers.

MOSS and WSS SP2 can break PDF searching.

notorioustech — Tue, 28 Jul 2009 14:01:00 GMT

It seems that MS only wants us to use the Adobe ifilter as SP2 actually hardcodes in the GUID for the Adobe ifilter.

http://support.microsoft.com/kb/944447

NOTE: It does not even verify if you have that ifilter installed.

SO if you are using the modified Adobe 8/9 reader ifilter, Foxit or PDFLib.com ifilters you will need to re-install those or manually correct the GUID entries.

For many of you this is new news and have been indexing right along since installing. This means that you also need to do FULL crawls of all your content to ensure that nothing was skipped.

Enterprise Search and Cloud Computing don't have to be mutually exclusive concepts

notorioustech — Mon, 22 Jun 2009 04:40:00 GMT

It seems everybody is moving to the cloud with their Enterprise applications. Some of the most popular systems are available as remote hosted solutions: SAP(mysap.com), Salesforce.com, MS Exchange Online, MS CRM etc. But as companies start to migrate to these remote hosted best of breed systems they start to face challenges with data visibility aka Knowledge Management. Searching and Reporting are critical tools for the the enterprise but if your data is no longer centralized how can you get a complete picture of all your data.

Enterprise Reporting has always been a critical part of business, being able to run reports and analyize data across multiple systems whether ad hoc or OLAP is critical for making reliable business decisions. But now building that data wharehouse is not so simple, you can't just import directly from your local databases of each of your systems. Hope is not lost though, I have researched many of those systems and there usually is a partner or two with an offering to provide that data extraction automatically for your reporting purposes. I would be curious to find out if anyone has build a centralized cloud based reporting solution that includes connectors for those system like SAP and Salesforce.com, if not i think i have my next startup company plan.

Anyways, this is a blog about search so let's focus on that. If you are hosting SharePoint internally (or FAST) for your Enterprise Search solution then how do you continue to maintain a centralized search interface when your data isn't so centralized anymore. The first option that comes to mind is Federated Search.

There has been alot of confusion about what Federated search means so let's clear that one up first. Federated Search is simply the combining of search results in a single interface from multiple search engines, for SharePoint that means when you run a query you get your regular search results in one webpart and completely seperate set from your Federated sources in another. In order for Federated search to work, either the vendor has to offer a standard Federatable (is that word?) interface or your have to write your own. There is possibly a second psuedo federated option to be considered depending on the remote systems search interface. Adding an iframe to host the remote search interface and some javascript to pass the search parameters has worked for me for some systems (sharepointsearch.com uses that model actually for it's search) and may provide what you need.

While i am at it, i might as well list the pros/cons of Federating search interfaces
Federated Search Pros:

Security can be handled by the remote system so no translation is required into your native search system (if you can resolve your single sign on issues)
No delays on index updates so search results are usually fresh
Minimizes network and internet bandwidth as the data doesn't have to be extracted out

Cons

Reliance on a Single Sign On model usually which adds an additional step before end users can search.
Loss of integrated features like Facetted navigation, customizable search results, and advanced searching
Complete loss of the relevancy capabilies of your search engine, your data is not actually in the index

You will have to be the judge on whether the loss of some search features is enough to justify the third option, index your cloud data with your Enterprise search engine. How you ask? Here are your options:

If you are already extracting out the data for reporting purposes into a datawarehouse most search engines provide a way to crawl sql data ( or there are vendors to help. See sharepointsearch.com for the list).
Write your own connector (see prior blog posts about how) to the remote systems APIs
Look to a third party vendor (again see sharepointsearch.com for a complete list).

NOTE: For Exchange, MS CRM, Salesforce.com/force.com and SAP Cloud systems see BA-Insight for SharePoint and now FAST. (shameless plug)

Technorati : cloud enterprise search fast sharepoint knowlege management sap salesforce.com
Del.icio.us : cloud enterprise search fast sharepoint knowlege management sap salesforce.com
Zooomr : cloud enterprise search fast sharepoint knowlege management sap salesforce.com
Flickr : cloud enterprise search fast sharepoint knowlege management sap salesforce.com

SP2 triggers SharePoint license expiration!

notorioustech — Fri, 05 Jun 2009 13:36:00 GMT

I don't know how i missed this post from a couple weeks ago, but asking around nobody seemed to have caught it so here it is.

http://blogs.msdn.com/sharepoint/archive/2009/05/21/attention-important-information-on-service-pack-2.aspx

Here is an excerpt

During the installation of SP2, a product expiration date is improperly activated. This means SharePoint will expire as though it was a trial installation 180 days after SP2 is deployed. The activation of the expiration date will not affect the normal function of SharePoint up until the expiration date passes. Furthermore, product expiration 180 days after SP2 installation will not affect customer’s data, configuration or application code but will render SharePoint inaccessible for end-users.

Here is the KB article on it: http://support.microsoft.com/kb/971620

Internet Explorer 8 causes 401 problems on SharePoint Servers using Host Headers. Don't install till you read this!

notorioustech — Wed, 20 May 2009 18:21:00 GMT

The issue appears when you use host headers in IIS to distinguish between your sites. If you try to open your NTLM based site locally it will return a 401 error after 3 tries to log in and any SharePoint webservices calls will fail also locally.

After installing IE 8 (or rather letting auto update run) I found that my SharePoint webservices calls were no longer working. I have webparts and sites that do webservice calls back to the same machine and have worked for 2 years without problems.

Here is the fix: (http://support.microsoft.com/kb/896861)

Method 2: Disable the loopback check
Follow these steps:

Click Start, click Run, type regedit, and then click OK.
In Registry Editor, locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
Right-click Lsa, point to New, and then click DWORD Value.
Type DisableLoopbackCheck, and then press ENTER.
Right-click DisableLoopbackCheck, and then click Modify.
In the Value data box, type 1, and then click OK.
Quit Registry Editor, and then restart your computer.

From the comments, it appears it may have been caused by one of the updates that happened at the same time.

Webinar: Connecting Enterprise Systems to SharePoint Search with Security: SAP, Exchange, Vault, Documentum, Hummingbird, MS Dynamics, Oracle, SQL Server and More

notorioustech — Thu, 16 Apr 2009 01:46:00 GMT

Enterprise Search means being able to search across all your enterprise systems, but that means you need to find a way to extract out that data and get it into your search engine.

One option is to use a 3rd party connector solution like BA-Insight's Longitude Enterpise Connectors.

See how BA-Insight can index any Enterprise System with Full Security mapping, you will also see a demo of their Longitude Search for SharePoint which provids facetted navigation, real time previews and more.

View the webinar recorded on April 17th:
https://www2.gotomeeting.com/register/588636416

Best Practices for LARGE SharePoint Lists and Documents Libraries

notorioustech — Wed, 08 Apr 2009 22:21:00 GMT

SharePoint lists and document libraries are not designed to handle large numbers of items, in fact as soon as you add more than a few thousand you will see a warning within the List Settings Page that says:

This list or library contains a large number of items. Learn about managing a large list or library and ensuring that items display quickly.

As part of my product suite I consistently have lists with hundreds of thousands of items in them and have learned many valuable lessons in making these kind of numbers workable.

Hopefully some of what i learned can help you.

1. First off do not use the DataView webpart (from the SharePoint designer) or any xml based rendering webpart as they first grab all the items in the list and then apply the XLST to render and you will face both memory and huge performance problems. The regular list webpart works fine even for large datasets, but you will not be able to get total list counts regardless.

2. If you are planning on sorting or filtering by any columns be sure to add them to the lists indexed fields in the list settings page.

3. SharePoint is a database driven application and the list data is thus stored in database tables. The way it stores them is very denormalized as each columns data is stored in a single row, which means if you have 2 list items with 3 columns you get at least 6 rows of actual sql data. This adds up very fast and will quicky cause both disk and index fragmentation. Always ensure that your logical disks are defragmented once a week and use the SQL Management studio to create an index maintenence plan for all your databases.

4. For Developers using the Object model:

a. Getting counts: never reference the list item collections count property as that will trigger a complete retrieval of all the items in the list and possibly throw memory errors. If you need the list count use list.count instead of list.items.count

b. Item Access: never access the list item collection by index position as it will re-retrieve each item when accessed.

for (int i=0;i< list.items.count;i++) // bad

SPListItem li = list.items[i]; //bad

instead use the standard enumeration model to walk though your collection when needed

foreach (SPListItem li in list.items) //good

d. Paging through large numbers of list items: use the SPQuery and the RowLimit property to page through your larger list collections

SPQuery query = new SPQuery()
query.RowLimit = 10000;
do
{

SPListItemCollection myItems = list.GetItems(query);

for (SPListItem li in myItems)
{
//do something here
}
query.ListItemCollectionPosition = myItems.ListItemCollectionPosition;

} while (query.ListItemCollectionPosition != null);

e. Deleting large numbers of list items (new): use SPWeb.ProcessBatchData whenever possible. Taking above advice here is my idea of the best model for item deletion.

SPList lst = SPContext.Current.Web["MYLIST"];
SPQuery query = new SPQuery();
query.ViewFields = "<FieldRef Name='ID'/>";
query.IncludePermissions = false;
query.RowLimit = 20000;

// first get all the id's to be deleted
ArrayList al = new ArrayList();
do
{
SPListItemCollection myItems = lst.GetItems(query);

foreach (SPListItem item in myItems) // note use enumeration
{
if (item.ID == skipId) continue;
al.Add(item.ID.ToString());

}
query.ListItemCollectionPosition = myItems.ListItemCollectionPosition;
} while (query.ListItemCollectionPosition != null);

StringBuilder sbDelete = new StringBuilder();
sbDelete.Append("<?xml version=\"1.0\" encoding=\"UTF-8\"?><Batch>");
string listguid = lst.ID.ToString();
int bcount = 0;

for (int i = 0; i < al.Count; i++)
{
if (bcount > 1000)
{
sbDelete.Append("</Batch>");
lst.ParentWeb.ProcessBatchData(sbDelete.ToString());
sbDelete = new StringBuilder();
sbDelete.Append("<?xml version=\"1.0\" encoding=\"UTF-8\"?><Batch>");
bcount = 0;

}
bcount++;
sbDelete.Append("<Method>");
sbDelete.Append("<SetList Scope=\"Request\">" + listguid + "</SetList>");
sbDelete.Append("<SetVar Name=\"ID\">" + al[i].ToString() + "</SetVar>");
sbDelete.Append("<SetVar Name=\"Cmd\">Delete</SetVar>");
sbDelete.Append("</Method>");
}

sbDelete.Append("</Batch>");
lst.ParentWeb.ProcessBatchData(sbDelete.ToString());

I hope this helps. If you have any further advice please comment on this post and i will incorporate it for everyone. THANKS!

Joel's list of more performance related information

also see http://go.microsoft.com/fwlink/?LinkId=95450&clcid=0x409 for an earlier white paper on the subject

SharePoint Indexing Performance Tuning Tips

notorioustech — Fri, 06 Mar 2009 19:50:00 GMT

There are many factors involved in the SharePoint crawling process that can impact indexing performance. There are also some steps you can take to improve that. Here are the common causes and their resolution:

Indexing Performace is set at reduced - common mistake on the configuration screen for the index service. See Central Administration > Operations > Services on Server > Office SharePoint Server Search Service Settings and set to Maximum.
Number of Connections - by default the indexer will run a limited number of simultaneous threads (6 usually) per host. This can be increased manually by adding specific Crawler Impact Rules for each host. You can really improve speed by setting a large file server up to 64 connections. This number is just a suggestion btw to SharePoint, it also looks at other factors like the number of processors (8 * #procs). And also watch your network for bottlenecks and those pesky RPC errors you may get in your logs (dial it back of you see those)
Crawled systems are slow or hosted on remote networks. - not a lot to be done here, except by moving those files closer.
Overlapping Crawls - SharePoint gives priority to the first running crawl so that if you already are indexing one system it will hold up the indexing of a second and increase crawl times.
- Solution: Schedule your crawl times so there is no overlap. Full crawls will take the longest so run those exclusively.
IFilter Issues - the Adobe PDF IFilter can only filter one file at a time and that will slow crawls down, and has a high reject rate for new PDFs
- Solution: Use a retail PDF filter from pdflib.com or Foxit
Not enought Memory Allocated to Filter Process - an aspect of the crawling process is then the filtering deamons use up to much memory (mssdmn.exe) they get automatically terminated and restarted. There is of course a windup time when this happend and can slow down your crawling. The current default setting is pretty low (around 100M) so is easy to trip when filter large files. You can and should increase the memory allocation by adjusting the following registry keys
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set DedicatedFilterProcessMemoryQuota = 200000000 Decimal
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set FilterProcessMemoryQuota = 200000000 Decimal
Bad File Retries - there is a setting in the registry that controls the number of times a file is retried on error. This will severly slow down incremental crawls as the default is 100. This retry count can be adjust by this key:
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Global\Gathering Manager: set DeleteOnErrorInterval = 4 Decimal
General Architecture Issues - Ensure that you have at least 2 Gig of free memory available before your crawl even starts and that you have at least 2 real processors available.
Disk Health - the nature of the indexing process causes extensive fragmentation of the file system for both the index server and the database server. Schedule defrags routinely and after all full crawls. Ensure you have enough diskspace always.
Run 64 bit OS - school is still out on this one, i personally haven't seen much difference as long as there is enough memory and the same processor types, but MS recommends this for large deployments.
Proper SQL Server configuration (new) - For large (>5 Million) item indexing you will need to plan ahead for the correct SQL Server configuration in order to scale to these numbers. There is one table in particular that grows at 40x the number of items and can severely hinder peformance if you do not treat your SQL Environment like a very large data wharehouse. Here are my recommendations based on experience:

1.       Raid 10 Direct Attached Storage Only – minimum 4 arrays – 16 disks

2.       Multiple File Groups- pre-allocate all database files and partition on dedicated separate arrays and assign 1-1 for:

a.       Indexes for SharedServices1_search db

b.      Temp and system databases/tables

c.       transaction log for SharedServices1_search db

d.      Table content for SharedServices1_search

                                       i.            For every 5 million items have additional file from dedicated drive in dedicated file group. Content and load is spread across and will improve performance

   3. When intially crawling, be sure to pause your crawls every day or so and rebuild/reorg the indexes on the SharedSevice1_search_db database (especially the indexes on MSSDocProps table)

NEW FROM MS: http://technet.microsoft.com/en-us/library/cc298801.aspx Whitepaper on just this topic!!!!

NOTE: It is a good idea to open up perfmon and look at the gatherer stats while indexing. There is a statistic called Performance Level and this reflects the actual level that the indexer is running at where 5 is max and 3 is reduced. Even if you set everything to max the indexer may decide to run at reduced anyways based an some unknown factors.

This is a good read too: http://technet.microsoft.com/en-us/library/cc262574.aspx (Estimate performance and capacity requirements for search environments)

Addendum (7/28/2009)

Here is another good read from MS (http://technet.microsoft.com/en-us/library/cc850696.aspx (Best practices for Search in Office SharePoint Server)

Searching Lotus Notes Data Securely with SharePoint

notorioustech — Tue, 24 Feb 2009 17:23:00 GMT

SharePoint has built in functionality to index Lotus Notes content, but unfortunatly it has many shortcomings for the Enterprise. Before you exhust many cycles trying to get the built in connector working, you should educate yourself on all the features and functionality to determine if it meets your needs.

BA-insight has released a connector that meets all of the customer demands for searching notes. Some key points are that they fully honor security from group level security, to filters, to reader fields which the MS one does not. They also give granular control of the metadata mappings like title.

See a comparison full comparison matrix: http://www.ba-insight.com/collateral/notesfeaturematrix.pdf

Their UI product give an enhanced view of the Notes results also, by expanding out the list of attachments and showing a real time preview of each one.

They are giving on online webinar this Thursday (2/26/2009) at 12 noon EST which is open to the public: https://www2.gotomeeting.com/register/730057510

Request more information: Online Request Form

Microsoft plans for FAST finally revealed

notorioustech — Tue, 10 Feb 2009 19:07:00 GMT

Read the press release here:

http://www.microsoft.com/presspass/press/2009/feb09/02-10NewEnterpriseSearchPR.mspx

Basically in Office 14 Fast will be included with the Enterprise CAL (with some additional costs for servers and most likely some consulting requirements).

More importantly they are not ripping out the current platform search bits, so there should be a straightforward upgrade path from MOSS 07 and no loss of investments in current technologies.

BA-Insight's SharePoint products for Search fuel growth in the down market

notorioustech — Wed, 04 Feb 2009 16:18:00 GMT

While many businesses are scaling back and shedding resources in response to this global economic crisis, BA-Insight proves that success can exist even in the worst of times. They continue to grow and expand, adding international offices and resources. See http://arnoldit.com/wordpress/2009/01/31/ba-insight-triples-in-2008/

It makes sense though, organizations, fueled by the need to make the most of their existing infrastructure and assets, are beginning to realize that they already own an enterprise class search system. Why struggle with the additional hardware, training, and costs of Autonomy, Fast, Coveo and others, and then still need to figure out how to integrate with your SharePoint portal?

A high end UI and connectivity to existing Line of Business systems are the only missing components to complete the SharePoint search solution and this is what BA-Insight provides.

Contact BA-Insight for more information about their offerings at http://ba-insight.net or directly submit a request for information or quote here: Online Request Form

Longitude Enterprise Connectors for MOSS and MS Search Server 2008

Security: All of the connectors fully map and honor security of the source system with Full AD mapping and synchronization.

Extensibility: Pull and combine metadata from multiple systems to extend your searchable record

Control: Wizards and a flexible administration site provide granular control of indexing, including ability to format and replace metadata on the fly.

Enterprise Connector Listing:

· Exchange Mailbox Stores – for desktop search replacement or discovery.

· Documentum – now you can retain your full security mappings.

· Lotus Notes – flexible and fully secure alternative.

· Hummingbird – highly customizable.

· WorkSite – documents and metadata combined with security.

· SAP – yes, you can now index SAP data securely.

· Enterprise Vault – unlock your archives to Enterprise Search.

· Interaction – legal CRM.

· Custom SQL Database Systems – define your own connectors to SQL Server or Oracle with simple SQL wizards. Leverage full security mapping capabilities and include linked files, or even file blobs, in your database.

· Custom API based Systems – use the Web Services connector to wrap any API based system.

· Releasing Soon: Salesforce.com, MS Dynamics, MS CRM, FileNet, Stellent.

· 3-5 day turnaround for new connectors. So if you don’t see yours, just ask for it!

Longitude Search UI for MOSS and MS Search Server 2008

Once you see this search center replacement in action, you and your users will want it. Once you use it, you can't imagine searching any other way. Real time document previews save so much time searching, you know right away if it was the result you wanted. The facetted navigation, while similar to the codeplex variety is built into the product. Ask for demo and find out more, there are search tag clouds, automated relevancy improvements based on user actions and parametric searching.

Elements of a Successful SharePoint Search

notorioustech — Wed, 17 Dec 2008 16:53:00 GMT

Mark Gerow has written a great article on the practical aspects of implementing SharePoint search. Great Reading.

http://www.law.com/jsp/legaltechnology/pubArticleLT.jsp?id=1202426774552#

Here are some of his topics:

WHY SEARCH IS DIFFERENT?

Why can't it just work like my favorite Internet search engine?

How do I know others won't be able to find my secured documents?

Does it search my Inbox? ( yes it can btw with BA-insights.com http://www.ba-insight.com/exchange.html connector).

Do I have to stop searching the old way? I liked it.

OF PROTOCOL HANDLERS, IFILTERS AND THE BDC

CONCEPTUAL VS. KEYWORD SEARCH

FACETED SEARCH