Michael Bergman, federated search luminary, just wrote on the subject, preferring the term “data mixing.” He explains the concept:

What is Data Mixing and Why is it So Hard?

As a new term there is no “official” definition of data mixing. However, I think we can consider it as generally equivalent to the older data federation concept.

Data federation is the bringing together of data from heterogeneous and often physically distributed data sources into a single, coherent view. Sometimes this is the result of searching across multiple sources, in which case it is called federated search. But it is not limited to search. Data federation is a key concept in business intelligence and data warehousing and a driver behind master data management (MDM).

Bergman explains that data federation was a hot research topic in the 1980s. Computers of different hardware, operating systems, databases, and other software were proliferating. Today’s robust and ubiquitous networking protocols were far from mature then. There were no dominant standards for data representation in the 80’s. Today we take interoperability for granted; if two systems don’t speak to one another directly we expect that someone has already developed software to bridge the gap. The whole Internet speaks TCP/IP. XML is everywhere.

So, we can say that it took the solving of some major data federation problems to lay the foundation for the Internet and the Web that we enjoy today.

Bergman further explains that the next major challenge is in semantics:

The Internet and its TCP/IP and Web HTTP protocols and XML standards in particular, have been major contributors to overcoming respective physical and syntactical and data exchange heterogeneities. The current challenge is to resolve differences in meaning, or semantics, between disparate data sources. Your “glad” may be someone else’s “happy” and you may organize the world into countries while others organize by regions or cultures.

I recommend Bergman’s article, especially if you have an interest in the Semantic Web. It’s moderately technical but it’s worth the read to understand where data federation fits into the Semantic Web.

ShareThis

My concern was, and is, that libraries and research organizations would consider giving away their responsibility to select quality sources for their patrons for what I imagine to be two primary reasons: (1) library patrons don’t like to wait 30 seconds for federated search results, and (2) (possibly) cost savings. I don’t have a lot of sympathy for the Google generation. Even though I’m an American and my culture has taught me that immediate gratification is a good thing I think 30 seconds is a small price to pay to see better results. Cost I can’t speak to as I don’t have any figures.

One of my colleagues pointed me to an article by scientist and writer Michael Nielsen, Is scientific publishing about to be disrupted?, which only strengthens my belief that access to content from aggregators only supplements access via other methods such as federated search.

Michael Nielsen is a very accomplished scientist. His bio lists some of his impressive credentials:

Michael Nielsen is one of the pioneers of quantum computation. Together with Ike Chuang of MIT, he wrote the standard text on quantum computation. This is the most highly cited physics publication of the last 25 years, and one of the ten most highly cited physics books of all time (Source: Google Scholar, December 2007). He is the author of more than fifty scientific papers, including invited contributions to Nature and Scientific American. His research contributions include involvement in one of the first quantum teleportation experiments (related), named as one of Science Magazine’s Top Ten Breakthroughs of the Year for 1998, quantum gate teleportation, quantum process tomography, the fundamental majorization theorem for comparing entangled quantum states, and critical contributions to the formula for the quantum channel capacity. A full list of papers is here.

Nielsen’s article argues that there is impending disruption of scientific publishing. The article is fascinating, Nielsen is a compelling and well-informed writer and I recommend you read the fairly long article and, if you have time, that you follow at least some of the numerous links. I want to also add that I had the opportunity to spend some time with Nielsen at a conference he helped to organize at the Perimeter Institute and I very much appreciate how incredibly down to earth the man is.

What I found most valuable in Nielsen’s writing were various examples of science being published in non-traditional ways.

One example is Nielsen’s response to a New York Times editorial about the death of newspapers. Here’s a snippet from the editorial:

There’s a great deal of good commentary out there on the Web, as you say. Frankly, I think it is the task of bloggers to catch up to us, not the other way around… Our board is staffed with people with a wide and deep range of knowledge on many subjects. Phil Boffey, for example, has decades of science and medical writing under his belt and often writes on those issues for us… Here’s one way to look at it: If the Times editorial board were a single person, he or she would have six Pulitzer prizes…

And here’s Nielsen’s poignant response:

[The New York Times editorial piece] demonstrates a deep commitment to high-quality journalism, and the other values that have made the New York Times great. In ordinary times this kind of commitment to values would be a sign of strength. The problem is that as good as Phil Boffey might be, I prefer the combined talents of Fields medallist Terry Tao, Nobel prize winner Carl Wieman, MacArthur Fellow Luis von Ahn, acclaimed science writer Carl Zimmer, and thousands of others. The blogosophere has at least four Fields medalists (the Nobel of math), three Nobelists, and many more luminaries. The New York Times can keep its Pulitzer Prizes.

Nielsen’s point is clear. The blogosphere is a tremendous resource to scientists. Libraries and research organizations miss huge amounts of valuable and current resources if they only provide access to content from major publishers (or their aggregators.) I do realize that the writings of probably all of the bloggers that Nielsen mentioned is available through Google and might not make sense to federate. The problem with searching Google for excellent science is that you need the time and discernment to find the good stuff. But, however one might access science content, the power of traditional publishers is waning which is a really good reason to not depend on them for all the science worth reading.

Here’s another excerpt from Nielsen’s article, this one on innovative ways to communicate science that are sprouting up everywhere:

What’s new today is the flourishing of an ecosystem of startups that are experimenting with new ways of communicating research, some radically different to conventional journals. Consider Chemspider, the excellent online database of more than 20 million molecules, recently acquired by the Royal Society of Chemistry. Consider Mendeley, a platform for managing, filtering and searching scientific papers, with backing from some of the people involved in Last.fm and Skype. Or consider startups like SciVee (YouTube for scientists), the Public Library of Science, the Journal of Visualized Experiments, vibrant community sites like OpenWetWare and the Alzheimer Research Forum, and dozens more. And then there are companies like Wordpress, Friendfeed, and Wikimedia, that weren’t started with science in mind, but which are increasingly helping scientists communicate their research.

These Web 2.0 science offerings, at least the ones that provide an API or other mechanism for efficient search, are prime candidates for federation as they constantly generate new content.

One last quote from Nielsen. I very much enjoyed the great examples Nielsen packed into this paragraph of outstanding science being found in blogs of all places.

It’s easy to miss the impact of blogs on research, because most science blogs focus on outreach. But more and more blogs contain high quality research content. Look at Terry Tao’s wonderful series of posts explaining one of the biggest breakthroughs in recent mathematical history, the proof of the Poincare conjecture. Or Tim Gowers recent experiment in “massively collaborative mathematics”, using open source principles to successfully attack a significant mathematical problem. Or Richard Lipton’s excellent series of posts exploring his ideas for solving a major problem in computer science, namely, finding a fast algorithm for factoring large numbers. Scientific publishers should be terrified that some of the world’s best scientists, people at or near their research peak, people whose time is at a premium, are spending hundreds of hours each year creating original research content for their blogs, content that in many cases would be difficult or impossible to publish in a conventional journal. What we’re seeing here is a spectacular expansion in the range of the blog medium. By comparison, the journals are standing still.

At SLA 2009, Abe delivered a presentation: A Journey to 10,000 sources. The talk was about (this blog’s sponsor) Deep Web Technologies‘ efforts to search initially hundreds, then thousands, and eventually 10,000 sources. The accompanying paper makes this important argument for making a wider range of science information available to researchers:

By relying on only the content available from the major publishers and aggregators, researchers miss other important content, in particular the output of scientists who do not publish in mainstream journals. The world is shrinking, the brain pool is growing, and the output of science is everywhere.

While one may argue about the merits of federation vs. crawling and indexing vs. discovery services those arguments frequently focus on the technological merits of particular approaches. The more important question, I think, is what information is worth your while to see? For most of us that information can’t all be federated, or all indexed, or all provided to us by a discovery service. I think federated search will continue to evolve into this hybrid being where multiple technologies are enlisted to give scientists what they need.

ShareThis

When I learned about About.com’s $18 guide to Online Research I became very curious to see if I had found a complement to Going Beyond Google. I got a review copy from the publisher and what follows are my impressions of the book.

The Online Research book is authored by Wendy Boswell, About.com’s guide to Web Search. The book is 276 pages long and has 15 chapters plus several appendices. The book was published in 2007. While this may seem pretty current, depending on what month the book was published it might be two and a half years old. That’s getting old given the numerous references to web resources.

My main interest was in the value of the book for proselytizing about the value of federated and deep Web searching. Chapters 8, 9, and 10 were most relevant:

Chapter 8: Digging Deeper with the Invisible Web
Chapter 9: The Web as Your Personal Librarian
Chapter 10: Evaluating Web Sites for Credibility

For the sake of completeness I’ll list the other chapters although I only skimmed them:

Chapter 1: An Introduction to the World Wide Web
Chapter 2: The Basic Web Search Toolbox
Chapter 3: Using Search Engines
Chapter 4: Google Tips and Tricks
Chapter 5: Searching the Web with RSS
Chapter 6: The Niche Web
Chapter 7: Using the Social Web in Searches
Chapter 11: Finding Multimedia on the Web
Chapter 12: Mining the Blogosphere
Chapter 13: Keeping Your Web Searches Private
Chapter 14: Most-Requested Reader Searches
Chapter 15: Web 2.0

Chapter 8: Digging Deeper with the Invisible Web

This chapter provides a really good introduction to the deep Web. I particularly appreciated this paragraph:

Why is the invisible Web important? I can answer that in one word: quality. Most of the information on the invisible Web is very topic-focused, simply because most of this fantastic information is packaged in various databases concerning everything from archeology to zoology. Because this information is so narrow - and for the most part, academically oriented - you’re more likely to obtain higher than average quality search results in a shorter amount of time, which definitely comes in handy when you’re trying to do a research paper on a deadline.

Bingo! I couldn’t have said it better. I like the author’s clear and simple style of writing. She goes on to discuss the size of the deep Web, citing statistics from Michael Bergman’s Bright Planet seminal paper on the subject. She explains how crawling differs from deep Web searching and how “invisible Web gateways” provide access to deep Web content. Most of the rest of the chapter lists deep Web resources (portals and search engines.)

I learned a handy trick for finding deep Web databases in this chapter. Add the word “database” to your queries. Sure enough, when I tried the example of searching for the two words “flowers” and “database” (not as a phrase) the top few results were all to searchable databases of flower-related information. I found a pressed flower database, a gardening plant finder from the BBC, and a searchable database of companies in the flora industry, to name a few.

Chapter 9: Using the Web as Your Personal Librarian

This chapter is about finding a topic to research. It provides more web resources; these are general reference resources intended to get a researcher high level information about a subject. Some of the resources are deep Web ones: The Arts Database at Yale, SciSearch, Science.gov, and Biography.com are just a few of the ones mentioned. The chapter does touch on how to evaluate resources for credibility but leaves the deeper discussion for the next chapter.

I have to note that this book is chock full of resources. I find this chapter, like much of the book, is filled with descriptions and URLs to many great web sites. While I like the level of detail I also find it a bit overwhelming. Too many sources and not enough time to discover them all. That’s how I feel. So, I find myself skimming much of the book, looking for what’s relevant to me. Maybe that’s the author’s intent.

Chapter 10: Evaluating Web Sites for Credibility

Of course, with the scholarly federated search engines out there one needn’t worry about the credibility of information. It’s when one strays from the deep Web search engines that one has to worry about the credibility of the content found. I do think that this question of credibility is a critical one, especially for researchers. But, even the public should be more concerned about what’s true. Just because it’s online doesn’t make it true, right? So, how do we know what’s true?

This chapter considers factors that determine credibility: outside editorial oversight, double-checking of facts, and maintenance by trained experts. Specific advice is provided on how to evaluate a web-site:

• Who’s in charge?
• Is it absolutely clear which company or organization is responsible for the information on the site?
• Is there a link to a page describing what the company or organization does and the people who are involved (an “About Us” page)?
• Is there a valid way of making sure the company or organization is legit - is this a real place that has real contact information?
• Is the site telling me the truth?
• What is your source really trying to tell you?

I appreciate this exploration of critical thinking skills. These skills are not ones I hear discussed particularly often. As young people enter their college years, given how much time they’re going to be spending online, I think it’s important that they learn to filter what they read.

What do I think of the book? Do I recommend it? I like it and I recommend it. It is certainly not a replacement for Going Beyond Google. It is not an academic book. I wouldn’t use this as the only book in a college course but I would use it as a second source. The book provides a very readable introduction to the deep Web. It provides too many resources but you may not find that overwhelming. It gives a really great introduction to web searching, which applies to federated search as much as it does to searching the crawlers. This is a great book to give to a child heading off to college, especially if the child has an aptitude for or an interest in information science.

ShareThis

Michelle asked me to moderate the panel, an honor I was grateful to accept. Michelle and I worked together to identify and recruit panelists and to define the topic. Here’s what we came up with:

Federated Search: A Wonder or a Waste?

Thursday, November 19, 2009
10:45 am – 11:30 am

Moderator: Sol Lederman, Federated Search Blog
Dr Peter Noerr, CTO, MuseGlobal, Inc.
Carl Grant, President, Ex Libris North America
Christopher Cox, Dean of Libraries, Western Washington University

Opinions about the value of federated search vary widely. Some view it as the optimal way to discover unified content. Others believe it to be a slow and poor substitute for searching the underlying sources. Some see it as a necessary evil and learn to tolerate it within their organizations. Come listen to our three panelists discuss their experience with numerous federated search deployments. Learn about the benefits federated search can offer as well as its shortcomings and common pitfalls. Walk away knowing if federated search is right for your organization and how to get from a blank search screen to one that delivers the results your users need.

Regular readers of this blog should recognize all three panelists.

Peter Noerr, CTO of MuseGlobal, has been a great supporter of this blog through a number of insightful comments although I’ve not yet corralled him into being a guest writer. MuseGlobal does quite a bit of content integration work that they don’t label as federated search. Peter was also one of the judges in our first-ever predict-the-future-of-federated-search writing contest.

Carl Grant, President of Ex Libris North America, has written a number of very well received guest articles for this blog. I’ve given Carl his own category in the blog to make it easy for readers to find all his articles. Carl and I first “met” when I wrote an article in January of 2008 about his old company, CARE Affiliates, and he responded. Carl was also a judge for the writing contest.

Christopher Cox, Dean of Libraries at Western Washington University, was the editor of the book “Federated Search: Solution or Setback for Online Library Services,” which contains a number of essays related to all stages of the federated search exploration, procurement, installation and test process. The book’s publisher kindly gave me three copies of the book which I gave away in exchange for reviews of parts of the book.

Our panel is going to discuss the different views about federated search. We’ll consider why some people absolutely abhor federated search while other love it. There might even be a middle ground.

If you’re planning to be at Enterprise Search Summit West come to our panel as I’d love to meet you. If not, then look for my blog article after the fact.

ShareThis

But federation is no panacea, at least as it is implemented today. A federated search application brokers a query, sending it to multiple search providers (i.e., the search interfaces to a variety of content repositories), whose results it then attempts to assemble into a coherent whole. Unfortunately, since most search providers provide little more than the top-ranked result pages, federated search applications are largely reduced to assembling a unified ranking of those disparate result pages.

This functionality is significant, and I do not mean to dismiss it. But it is not enough. In particular, this approach to federation necessarily assumes a lowest common denominator of search functionality–a consequence of the requirement to evenhandedly broker among a variety of search applications that vary in the richness of their APIs.

Note my emphasis of the phrase “lowest common denominator.” Peter Noerr, Chief Technology Officer for MuseGlobal, left a detailed comment which reads, in part:

This assumption of [lowest common denominator] LCD searching is taken as an obvious truism. But why?

If federated search systems are presumed to be capable of handling multiple record formats for data extraction from retrieved records, then why should they not be considered capable of generating Source specific search statements?

The reason (as for federated search itself) seems to be that most don’t because it is yet another messy thing to deal with on a Source by Source basis. Note that here I am talking about more than adding blanks and quotes to a search statement.

Peter raises an excellent point. There is a prevalent myth that federated search applications search as poorly as their most simple-minded source. But the myth makes no sense. If one of a dozen sources doesn’t allow an author search then the LCD myth implies that you’d get no author results from any source. Taken to its logical conclusion, given enough sources, your users would only ever get titles and URLs returned because, for any set of searchable fields, some source will fail to support one or more of them.

I suspect that what some people consider to be LCD behavior is that if, for example, you do an author search against a source that doesn’t have a searchable author field that you’re going to get nothing back from that source. As unpleasant as it sounds, depending on the source, it might be the right behavior, i.e. return nothing rather than return irrelevant results. In other cases it might be better to perform a full text search against that source than to not search the source at all if the user enters text into the advanced search author field. The whole issue of what is the lowest common denominator is a messy one because it’s not clear what the right behavior is. So, it’s not fair to say that federated search engines do the “wrong thing” when searching multiple sources.

Assuming that we agreed that LCD meant suboptimal behavior by the federated search engine then it’s certainly NOT true that federated search is forced to do LCD. Author search is a great example of how a federated search application can do much better than LCD. Author search is a pain. There are a variety of formats that a source could provide for specifying the author name. The source could expect LASTNAME,FIRST NAME (with or without the comma) or FIRSTNAME LASTNAME. Then there are first and middle names and first and middle initials to deal with. There are many ways a user could enter a name and many ways the source could want it. Plus, in some cases, a source could recognize more than one name format. A human could want to search for A S EINSTEIN, ALBERT S EINSTEIN, A SCIENTIST EINSTEIN, ALBERT SCIENTIST EINSTEIN, ALBERT EINSTEIN, or simply EINSTEIN. What happens if the source expects A EINSTEIN and you search for ALBERT EINSTEIN? Will the source do the right thing? Maybe. Maybe not. Smart connectors deal with these kinds of issues by translating, or mapping, the user’s search terms into a form that will yield the best results from a particular source. A smart connector would turn ALBERT EINSTEIN into A EINSTEIN just for the one source that needed that in order to give relevant results.

The reality to replace the LCD myth is that not all federated search engines are created equal and that some deal better with picky source behavior than do others. When I worked full time for blog sponsor Deep Web Technologies I dabbled in connector building and I worked closely with their connector developers on some projects. The Deep Web connectors have remarkably complex logic for what I used to think was a simple task. Dealing with search syntax, phrases, wildcards, booleans, and a host of other factors is far from trivial. The Deep Web connector developers put a lot of sweat and testing into each connector they build.

Connector quality is not the whole LCD story. Sometimes a source will return very few results compared to other sources. This puts the source at a disadvantage because the more results you have from a source the better the relevance ranking you can perform. A smart connector can try to get multiple results pages from that source. A source may be slow to return any results. Rather than ignore results from that source a smart federated search engine can initially show results from sources that did respond quickly and then update the results to merge in the late arrivals. Another major problem for federated search applications is to perform good relevance ranking when a source returns no snippet, or just the beginning of the abstract instead of a context sensitive snippet.

There’s more to the LCD story. Rather than repeat it all here I recommend you read a fairly in depth article I wrote a while back on what determines the quality of search results. I also recommend a white paper I wrote that distills the “quality of search results” ideas into four pages.

Hopefully, the next time someone tells you that federated search is confined to delivering the lowest common denominator of results you’ll be able to tell them that it “ain’t so” and you’ll be able to tell them why.

ShareThis

To be honest, warning lights went on in my head when I noticed articles in the SEO community about federated search. My knee jerk reaction was that the articles were going to tell people how to exploit federated search engines to get spam into the search engine indexes. The articles turned out not to be spammy at all but I still didn’t have any thoughts on what I might be able to contribute about SEO and federated search. The federated search applications I write about all provide scholarly content and getting your site aggregated by such an application is next to impossible, unless, of course, you’re a university or other major institution that provides top notch content. So, I mentally filed the two articles away.

Then, this morning I did my regular twitter search for “federated search” and noticed a bunch of tweets for the “Did You Know?” article. And, I noticed that the “federated link building” article was also getting attention on twitter. So, I reconsidered my attitude about these two articles. A number of people found these articles valuable enough to retweet. Maybe I could find something to say.

[begin tongue-in-cheek]
Heck, if nothing else, maybe I’d get some inbound links for this post. Maybe if I write about the subject and reference my article on enough blogs that discuss SEO and “federated search” I could get some new blog subscribers and new twitter followers. Maybe some of the blogs I could comment on would be “do follow” blogs with a high Google PageRank that would give me some PageRank juice, although with a PageRank of 7 I’m much more likely to give more through my link love than I would receive.
[end tongue-in-cheek]

Seriously, though, the “Federated Link Building” article got me thinking about SEO and federated search applications. There’s actually a tremendous interest in SEO by the organizations whose content is being federated. OSTI, one of my clients, makes a large amount of content available via OAI-PMH, through OAISTER and OpenDOAR, and to anyone else who wants to harvest it. Plus, they create large sitemaps to make it easy for the big three (Google, Yahoo!, and MSN) to index their content. This blog’s sponsor, Deep Web Technologies, is creating a product that creates topic pages which allow organizations to create static SEO-friendly pages that combine search results from their primary sources with reference material and images from outside sources. Scitopia.org is just one Deep Web Technologies customer that is very interested in the outreach potential of having Google find and index their content. Creating sitemaps and topic pages and making content “harvestable” are all ways to expose content in the deep web that is very difficult if not impossible for the crawlers to find without help. At the end of the day, every online organization wants to attract readers, followers, subscribers, and buyers of what they’re providing. Even organizations who give away information are wanting sufficient traffic to justify their worth.

Going back to Eric Ward’s “federated linking” idea about building links by getting aggregated by metasearch engines, I think that can work for folks who have authoritative, or at least highly desirable, content. This is a case of the rich getting richer; if you already have popular content then you may be able to attract search aggregators who might make your content even more attractive. If your content is poor then no aggregator with a reputation to maintain would want to touch it.

I think the point of Eric’s article might be that SEO folks don’t swim in the “deep web” waters and that they should consider dipping their toe in the water, so to speak. Many might never have heard of the deep web or think about how many metasearch engines there are out there. Charles Knight and his writers cover the alternative search engine industry at AltSearchEngines, including metasearch engines, so I recommend that blog as a way of looking for sites that might aggregate you.

My suspicion that many in the SEO world may never have heard of federated search is confirmed by the “Did You Know” article I mentioned early on. Author Catherine Potts leads with this:

What is Federated Search?

I really feel like an idiot because I was unaware of this term until yesterday but I guess it’s a pretty obvious thing. Eric Ward discusses it in his latest article over at Search Engine Land. After all the reading I’ve done… was I not paying attention or is this topic just not really something most SEOs focus on? It’s just not something I’ve had to really deal with in my job. It is something I’ve used while in college, however.

Catherine goes on to give a nice introduction to federated search and to the limitations of crawling and is kind enough to reference my blog. Plus, given the number of retweets by link builders, her thoughts and her article have struck a chord in her community.

So, those are my thoughts on SEO and federated search. If you like this article please Digg it, tweet it, Stumble it, and link to it from your highest ranking site. Oh, and if someone can tell me how I can get higher in the SERPs for “federated search” than Wikipedia I’d very much appreciate it.

ShareThis

ScienceResearch.com debuts as the most comprehensive Deep Web science search engine

Blog sponsor Deep Web Technologies announced the official launch of ScienceResearch.com. The science research portal itself is not new. What’s new is that it now searches over 400 sources in real-time. It runs inside an Amazon Cloud to quickly address changes in demand. Amazon’s Web Services Blog reported on the launch. Paula Hane at Information Today also covered the relaunch in a good amount of detail. All of this press coincides with Deep Web Technologies’ founder Abe Lederman’s presentation today at SLA: Journey to 10,000 sources. ScienceResearch.com is a key accomplishment on the road to searching 10,000 sources at once.

Scitopia.org adds new content partners

Scitopia.org is a federated search application, powered by search technology from Deep Web Technologies. The portal was created by leading scientific and engineering societies. Today they announced that three societies had joined. Scitopia boasts access to “over 3.5 million documents, plus patent and government data.” Content is from “twenty one societies spanning 350 years of sci-tech scholarship.”

OSTI web-site sports a new look and releases two new videos

One of my clients, the DOE Office of Scientific and Technical Information (OSTI), recently updated their web-site. I really like the modern look and the move to highlight RSS, their blog, alerts, and other user-friendly features. SLA’s Government Information Division Blog briefly covered the new look. OSTI, a pioneer in bringing federated search to the Federal government, is wholeheartedly embracing Web 2.0. On a somewhat related note, I recently reported on two videos that OSTI had released on their new YouTube channel. OSTI now has posted two more videos; these are with OSTI’s Director, Dr. Walt Warnick. They’re short and fun:

I bet you didn’t know this about searching!

The thrill of working for OSTI

ShareThis

This article addresses a limitation with federated search that could be removed if content sources provided specific metadata to federated search engines to improve relevance ranking. Good food for thought. ]

Daniel Tunkelang is the Chief Scientist and a co-founder of Endeca, a leading vendor of search technology. Before joining Endeca’s founding team, he worked at the IBM T. J. Watson Research Center and AT&T Bell Labs. Daniel pioneered the annual workshops on human-computer information retrieval and recently published a book on faceted search. He blogs at The Noisy Channel.

The problem with federated search

The case for federated search is straightforward: no single organization has all of the answers, and therefore no single index can ever hope to complete satisfy its users’ needs. Federation allows the developer of a search application to hedge his or her bets by bringing in knowledge from outside resources.

But federation is no panacea, at least as it is implemented today. A federated search application brokers a query, sending it to multiple search providers (i.e., the search interfaces to a variety of content repositories), whose results it then attempts to assemble into a coherent whole. Unfortunately, since most search providers provide little more than the top-ranked result pages, federated search applications are largely reduced to assembling a unified ranking of those disparate result pages.

This functionality is significant, and I do not mean to dismiss it. But it is not enough. In particular, this approach to federation necessarily assumes a lowest common denominator of search functionality–a consequence of the requirement to evenhandedly broker among a variety of search applications that vary in the richness of their APIs.

What I would like to see is federation of set retrieval, not just of ranked retrieval. At first glance, this aspiration may seem impractical, since the sets being combined are often too large to be aggregated at query time. I certainly don’t expect a federated search engine to dynamically aggregate gigabytes or even terabytes of documents to process each individual query!

Instead, we need search engines to support, as a standard API capability, the ability to return a summary of a set of search results. Faceted search has led the way in demonstrating the value of such summarization: it offers users a much richer overview of the search results than the users could hope to obtain from a handful of top-ranked results. For the same reason, it offers far more information to a federation broker.

Indeed, even faceted search is not enough. It is unreasonable to expect all search applications to use the same faceted classification scheme, and thus federators find themselves confronting the infamous vocabulary problem–perhaps more familiar to practitioners as an aspect of master data management. How do we address this problem?

With yet more summarization. For example, a facet value corresponds to the set of documents assigned that value, and thus to a distribution of occurrence frequencies on the other facet values. In fact, we can take this idea further and summarize a facet value in terms of a distribution on the words and phrases used in documents assigned that value. Given access to these summarizations, a federator can at least make educated guesses to establish correspondences and relationships among the facets returned by different search applications.

Of course, there is far more work needed to make such an approach effective and efficient enough to be practical. Summarization is more computationally intensive than simply ranking results, and combining summarizations is more complex than simply harmonizing relevance scores. But these are the challenges that represent the best opportunity to make federation a successful strategy.

ShareThis

Andy’s article does a nice job of explaining without overwhelming, and his short introduction skillfully avoids going into more detail than most people want. If you’ve ever wanted to explain (or understand) the connection between mashups and APIs, or how Twitter’s massive and rapid success can be attributed to embracing APIs, then this is the article for you.

APIs and Web 2.0 are intimately related. Tim O’Reilly, the man who is credited with coining the term “Web 2.0,” describes Web 2.0 as “the design of systems that harness network effects to get better the more people use them.” (See this article for a reference to this quote.) It is the proliferation of APIs, which facilitates the mashing up of services, that powers Web 2.0 and allows the rich interaction that makes the services grow, evolve, and improve.

APIs are at the heart of a number of technologies I’ve blogged about. Here are a few examples:

1. “Query-free” federated search. The prototype software that helps writers to find relevant content and references by watching what they type uses APIs to identify the right search terms and to perform searches of resources.
2. In A new paradigm for federated search, venture capitalist Steve Jurvetson’s distributed search paradigm relies on APIs to provide information about what pages users are browsing and to retrieve relevance information from the “Collective” database populated by others.

3. Federated search and information in context tells about new features of the Science.gov 5.0 release of the federated search application: clustering, new content sources, mashup of Wikipedia and EurekAlert results. Each of those new features relies on an API in one way or another.

APIs are absolutely everywhere! Federated search relies heavily on the technology. Searching sources, especially those that don’t have to be screen-scraped, uses APIs to specify how to search and how to retrieve results. AJAX is based on an API that specifies how to partially update a web page. Many modern web applications, not just federated search tools, use AJAX. If we consider that APIs are nothing more than protocols that allow two programs to communicate, even your humble web browser uses an API to get to your favorite federated search applications by talking to web servers using a standard called HTTP.

Can you think of other places where APIs are used with federated search? Leave a comment and let us all know.

ShareThis

The paper and talk consider what it will take to build federated search engines that simultaneously search, first one thousand, and eventually many thousands of sources. Here is the first paragraph from the paper’s abstract:

The future of scientific research depends on sifting through more information, more quickly, and more effectively. For a researcher to expect to be able to search 1,000 databases simultaneously for critical information is not unreasonable. While parallel search is the domain of federated search, the current paradigm has severe limitations. The limitations, which include speed, relevance ranking, and selecting the appropriate sources, become painfully obvious when one attempts to search more than a few dozen sources simultaneously. A paradigm is needed for scalability to not only overcome the limitations but also help us assimilate important information.

Check out all the contributed paper abstracts and come to Abe’s talk. And, visit the exhibitors, which include a few federated search vendors.

ShareThis