The commercial conditions for getting a set of records into an aggregated search environment seem to often be more onerous than allowing a broadcast search of the native source. Part of the problem lies in perception of the aggregated environment as being a competitor to the source of the records. This may be a true perception, a false one, or a wishful one.

The technology in this product does exactly what is described in Sol’s post - create searches from the typings of the user.

Another product performing the same functionality, though in a different use case, is gClick from DowJones. This provides contextually relevant company and people information for a retrieved web page.

Of course both of these are commercial products, so Andy would need to ask about how they can be used in a library web site.

I think the answer is a hybrid service. Index what you can (discovery service), federate what you can, harvest what you can, etc. A good federated search framework can integrate access to content that has been obtained by a number of different technologies.

In other words, lead with the question of what sources are important not with the question of what technology is the most convenient. Then find ways to get access to each of them. I’ve never said that federated search makes discovery services or crawling or harvesting obsolete.

With either broadcast search or an aggregated index, you only have access to what you have access to. And in either case, providing access to any arbitrary resource may or may not be possible, and will take a non-zero amount of ‘development’ time to add. With either case, the user using the tool only has access to searching content included in the tool, unless they leave the tool, which they can do in either case.

It seems to me you are assuming that broadcast search will have a lower barrier to including additional resources, and end up having more coverage? This to me is something that needs to be seen in practice, not assumed theoretically. The key thing will be comparing two actual tools, and seeing which tool includes more resources, and more relevant and quality resources, for some particular context (audience and use cases).

If SerSol’s aggregated index comes up short, then that will be to it’s detriment. But I don’t see how that can be assumed without an actual evaluation of two actual deployed tools side by side.

As the “odd one out” on sol’s panel (according to his post above), I should point out that my background is in libraries as well.

However, we have taken a piece of software (our Muse Platform) which powers a few library marketplace Federated Search systems, and expanded to pastures new. I’m not going to name names (look on our web site for those), but we have partners in publishing, medical information, enterprise software, corporate consultancy, and government systems. As you say federated access (not just search) is applicable across the board, and I hope to be able to bring some of our experience of that diversity to the panel.

I would also hope that the audience will keep us honest and make sure we keep the discussion to the fundamentals and not wander off into library minutiae.

While 2 of the 3 panelists are in the library space I believe that there will be applicability to the enterprise space. The enterprise search community needs to deal with many of the same issues as the library community - accessing subscription content, relevance ranking, ease of use, requirements gathering, installation, maintenance, and performance to name some of the major concerns.

The panel you are moderating at the ENTERPRISE SEARCH SUMMIT WEST seems to focus exclusively on the application of federated search for online library services. Of course, federated search has enterprise applications that go beyond online library services. Further, federated search has application beyond enterprise applications, such as enabling WorldWideScience.org to search the huge national science databases of over 50 countries around the world.

As your panel intends to debate whether federated search is “A Wonder or a Waste,” it seems inappropriate to limit attention on but one application of federated search.

How about helping me out with a panel to debate whether my Buick LeSabre is a wonder or a waste? The panel will limit attention to the application of my LeSabre for pulling stumps from my vacation mountain land.

Walt Warnick

As an operation I agree with the comments by Dave, Sol and Jonathan. LCD to me is literally where the Fed Search system issues a search of the same functionality (but possibly different syntax) to each Source. That implies that the functionality of the search is limited to that available from the least functional Source. I have deliberately used “functionality” to be as inclusive in the definition as possible. Indices, operators, relations, limits, even vocabularies are all examples of “functionality”.

Thus Dave’s A, B, C case is a perfect example of what I would call an LCD situation - in the second instance. Here the search is issued only as a full text search, because that is the only functionality supported by all three Sources. The other instance (where only A & B are searched) is dealt with below.

A couple of comments need to attach to this. Firstly it is interesting to ponder that the Fed search system must know enough about the different Sources to be able to determine what the “common” functions are. If it can do this, it is halfway(-ish) towards being able to handle Source Specific Searches (SSS). So why dumb down? Who knows? I don’t.

The second point is that the first of Dave’s instances (search A&B only) uses what we call a “strict” mode for the search. That is: if the Source can’t handle the search in it’s entirety, then fail for that Source, and return 0 results. This is not LCD, but rather a strange form of “HCF” (if we want to stick with basic arithmetic acronyms) where only those Sources which meet ALL the requirements (support all the functionality) are sorted. Different category, but a problem none-the-less for certain types of searches.

Where non-LCD searching is possible (SSS as used above) then it is possible to switch to a “relaxed” mode and allow some portion of the search to be processed and produce at least some results. (You want details - I knew you would. OK, the most obvious example is where a Source does not support a particular index and any terms for that Source are mapped to one it does support - in 99% of cases this is “keyword” or its functional equivalent. )

This mapped and relaxed operation does allow results from Sources which would be “0″s to a strict search and, in our experience, most people would rather get something back which is somewhat like what they asked for, rather than nothing. Often because they weren’t too sure about the query in the first place. Again a good result for some case, but not all. However take the union of the two cases and you cover what virtually everybody would want - and what you would get from the native Sources - always an important touchstone. (And this whole comment raise a host more issues for Sol to tease out into other posts - like the issue of notifying users of what the search has done.)

I think of the myth as the blanket statement that federated search engines rollover/suck because some sources are not easy to search well.

But, I’m interested to know what you all think LCD means. Abe and I discussed it for a while and realized that we weren’t quite sure what it means. So, please tell me.

At the same time there’s a different discussion about how some federated search engines handle tricky sources better than others. I’ll do what I can to find examples of smart connectors searching sources particularly well.