Avi Rappoport is an enterprise search consultant, helping companies improve search engine functionality for websites and intranets. She has a degree from UC Berkeley’s (then) School of Library and Information Science and spent 10 years in software development before becoming a search consultant. She is the editor of SearchTools.com and a frequent speaker and author, providing a strong focus on search usability in the broadest sense and sharing her conviction that search engines can always be better.

Avi created a web page with a summary of and links to a couple of versions of her presentation.

I greatly appreciate Avi’s consideration of the pluses and minuses of federation aggregation (i.e. discovery service) in a world that is often polarized about one approach being better in all cases.

My research for this presentation indicated that each is useful in specific circumstances (I know, no surprise there). Many data sources are obviously best accessed by one or the other, but it’s the corner cases that are tricky. Aspects to consider include:

• size of the content in the source
• how often your users need that content
• content change rate
• importance of real-time access control permissions changes
• content licensing rules
• available tools for indexing / querying
• difficulty of extracting and indexing
• quality of the internal search engine
• difficulty of sending queries and receiving results

The final slide has some sage advice:

Be open-minded, analyze the benefits of each approach for each data source.

One size does NOT fit all.

ShareThis

My mother died in March 2010 after a 15-year battle with Alzheimer’s, so I pay particular attention to news about this dreadful disease. A recent New York Times article caught my eye: “Sharing of Data Leads to Progress on Alzheimer’s.”

How did sharing data lead to progress on Alzheimer’s? A collaborative effort, the Alzheimer’s Disease Neuroimaging Initiative, was formed to find the biological markers that show the progression of Alzheimer’s disease in the human brain. The key was to share all the data, making every finding public immediately – “available to anyone with a computer anywhere in the world.”

Alzheimer’s research is an enormous task with limited returns. Dr. Michael W. Weiner of the San Francisco Department of Veterans Affairs said “Different people using different methods on different subjects in different places were getting different results, which is not surprising. What was needed was to get everyone together and to get a common data set.” Numerous entities were willing to shoulder the burden and work together on the project, sharing their information for the good of all.

According to Dr. John Q. Trojanowski, an Alzheimer’s researcher at the University of Pennsylvania, “It’s not science the way most of us have practiced it in our careers. But we all realized that we would never get biomarkers unless all of us parked our egos and intellectual-property noses outside the door and agreed that all of our data would be public immediately.” The National institutes of Health served as an “honest broker, between the pharmaceutical industry and academia.”

The effort has produced “a wealth of recent scientific papers on the early diagnosis of Alzheimer’s using methods like PET scans and tests of spinal fluid. More than 100 studies are under way to test drugs that might slow or stop the disease.” The collaboration has become a “model for similar efforts against Parkinson’s disease.”

This model matches OSTI’s mission. We share scientific data in an effort to encourage progress. In a recent interview with Juggle.com, Dr. Walter Warnick, OSTI Director, noted “It is common knowledge that science can advance only if it is shared. The OSTI Corollary to this is: Accelerating the sharing of scientific knowledge accelerates the advancement of science.”

That’s the idea behind ScienceAccelerator.gov – developed to advance discovery and to deliver science information.You can find information about current Alzheimer’s research by searching Alzheimer’s 2010 at ScienceAccelerator.gov.

Share the data!

ShareThis

When we deal with web scale data ‘discoverability’ of information is key. While ‘web search’ provides a lot of value today what we really need is to enable ‘data find data’. I like the differentiation in the book between ‘federated search’ and ‘federated fetch’. The latter needs adaptive systems that can discover new data correlations based on user context and new data collected.

This reference got me curious. Was the Web buzzing with discussion of federated search vs. federated fetch? Not exactly, according to Google, although there are 740 references to the phrase but only 24 of them are considered unique enough for Google to display. Interestingly enough, the first reference is to Jeff Jonas “When Federated Search Bites” article which I wrote about a month ago.

Once a directory reveals a pointer, you can go fetch it. Federated fetch does scale.

Google Books provides the term in the context of the Beautiful Data book:

So, federated fetch is the “end game,” if I understand the concept correctly. It’s what you get when, for example, a link resolver gets you to the full text copy of a book you can actually read.

There you have it, a new phrase I learned today.

ShareThis

Search Patterns is an absolutely outstanding book. I don’t get excited about search-related books very often but this one totally captivated me. O’Reilly sent me a review copy some months ago. It sat in a pile until I started seeing reviews and references to the book on the Web. The press prompted me to open the book.

The first thing I noticed in flipping through the book was the many high-quality color screen shots and illustrations. Plus, Search Patterns is printed on glossy paper to enhance the visual elements of the book.

At 173 pages (plus index) and a nice balance of text and images, Search Patterns is, at the surface, a quick read. But, there are numerous gems throughout the book so allow yourself plenty of time to read (and reread) sections that draw you.

I have to admit that I had a very difficult time planning for this review. That’s because I’m used to reading books that are very logical, very left-brained. This book speaks frequently in metaphor and connections between sections are not always clear. But, paradoxically, the writing and the logic are absolutely brilliant. As I hinted at in the previous paragraph I recommend you pick sections of the book you find interesting and read those sections as many times as you need to (along with related illustrations) to absorb the material.

Search Patterns has a preface and six chapters.

• Chapter 1: Pattern Recognition. Provides an introduction to the search problem.
• Chapter 2: The Anatomy of Search. Dissecting search into its components - users, interface, engine, content, creators, and context.
• Chapter 3: Behavior. A consideration of how users interact with search and how that should influence search engine design.
• Chapter 4: Design Patterns. A catalog of different mechanisms that support search - autocomplete, best first, federated search, faceted navigation, advanced search, personalization, pagination, and others.
• Chapter 5: Engines of Discovery. How content is organized.
• Chapter 6: Tangible Futures. Musings about the future.

My favorite things about the book are the many examples of search done right and also of search done poorly together with the many gems.

Chapter 1 has a great set of gems. Titled A Mapmaker’s Manifesto, this list of twenty items enumerates the authors’ beliefs and principles.

Zachary Spencer blogged some notes on a talk Morville recently gave. There are lots of gems in those notes.

Here’s a great interview with Peter Morville at JohnnyHolland.org.

This advanced praise quote by Dave Gray (founder and Chairman, XPLANE) sums up the essence of the book particularly well:

“It’s not often I come across a book that asks profound questions about a fundamental human activity, and then proceeds to answer those questions with practical observations and suggestions. Search Patterns is an expedition into the heart of the Web and human cognition, and for me it was a delightful journey that delivered scores of insights.”

I highly recommend Search Patterns to anyone who is designing a search engine, consulting on its development, or just wanting to understand why a particular search application is easy or hard to use. The book clarified so many issues I’ve seen (and continue to see) with search applications, federated or otherwise.

Finally, here are some of Morville’s thoughts on federated search, Google search, and the conversational search experience on YouTube:

ShareThis

The (Beta)DataFerrett helps you locate and retrieve the data you need across the Internet to your desktop or system, regardless of where the data resides.

DataFerrett is a unique data mining and extraction tool. (Beta)DataFerrett allows you to select a databasket full of variables and then recode those variables as you need. You can then develop and customize tables. Selecting your results in your table you can create a chart or graph for a visual presentation into an html page. Save your data in the databasket and save your table for continued reuse.

I have no idea how useful the tool is but their mascot sure is cute!

ShareThis

I certainly don’t know everything about federated search but I know enough to recognize what’s not federated search, at least not what most of us think to be federated search.

The article, really a rant, starts off reasonably enough:

Federated search: conducting a search against ?n? source systems via a broadcast mechanism without the benefit or guidance of an index.

I am speaking specifically about environments where the systems in the federation are heterogeneous, are physically dispersed, were not engineered for federation a priori, and are not managed by a common command and control system.

Here’s another reasonable statement:

Most organizations have some obligation to make sense of what they know. For example, the airline should know if the person added to the watch list is already an employee or already has a flight reservation. Ideally, the moment such facts become knowable, someone or some system should be notified. Think of this as ?the data speaks to itself.? I call this data finds data.

Yes, having new data trigger analysis is a good idea. But, IT’S NOT FEDERATED SEARCH.

So, the entire basis of the rant is that federated search is not this advanced analysis system I want therefore it sucks. That’s like saying that my oven doesn’t analyze the food I put in it and automatically cook it perfectly therefore my oven “bites.”

There may be a discussion about the challenges of analyzing federated data vs. indexed data but that has nothing to do with what federated search does.

What do you think? Does the article make sense to you?

ShareThis

OSTI has created a number of search portals (WorldWideScience.org, Science,gov, DOE ScienceAccelerator, DOE Energy Citations Database, and DOE Information Bridge to name a few) but few know about the history of the agency that created them.

OSTI grew out of the post-World War II initiative to make the scientific research of the Manhattan Project as freely available to the public as possible. On November 17, 1944, President Roosevelt wrote Vannevar Bush, then the Director of the Office of Scientific Research and Development, to request his counsel on how to capitalize on the experience of the United States’ R&D war efforts — most of which was done in utter secrecy — in the days of peace to come.

OSTI Director Dr. Walter Warnick tells the story of the development of OSTI, its role in advancing science, and how federated search serves that role in ways that Google can’t.

The paper, at 23 pages, covers the subject with a good deal of depth.

More than 60 years ago OSTI began advancing science.

Long before the Internet came along, OSTI advanced science by making research information widely available. OSTI annually responded to upwards of 50,000 requests for information and during the 1977 ?energy crisis? fielded more than 150,000 requests. OSTI operated one of the few federal printing plants in the United States, and in 1948 began an almost 30-year production of the world-famous Nuclear Science Abstracts, which greatly expanded access to nuclear science information. OSTI shouldered a lead role in providing materials to the Atoms for Peace Geneva Conferences, envisioned by President Dwight D. Eisenhower to pool nuclear information for sharing with peaceful nations. OSTI was instrumental in establishing the International Nuclear Information System (INIS), which promotes nuclear information exchange between 110 countries.

OSTI has capitalized on the power of the Web since the early days of the Web:

In 1994, OSTI created the first DOE home page, and it has made significant strides into the Information Age ever since, defining new electronic exchange formats, creating collections of digitized scientific and technical information, serving researchers directly, and developing an energy science and technology virtual library. OSTI today hosts three major collections of scientific and technical information: Science Accelerator, which features DOE R&D resources; Science.gov, which provides access to STI from federal science agencies through the U.S. government; and WorldWideScience.org, which offers resources from more than 60 nations around the world.

What’s next for OSTI? Here is one direction:

What is next? There is no inherent reason that a single tool cannot rely upon both a crawled index and a live federated search in parallel. Indeed, OSTI?s largest product does just that. It is the Eprint Network (http://www.osti.gov/eprints/). All in parallel, it searches 1.5 million eprints that have been crawled, plus an addition 5 million eprints hosted in 50 eprint databases, comprising in all about 100 million pages. As far as we know, there is no other tool in the world that virtually integrates such a quantity of eprints. Further, we are not aware of another publicly available search tool that searches federated databases and crawled indexes in parallel.

[ Disclaimer: I not only consult for OSTI but also for Deep Web Technologies who sponsors this blog and who built the search engines behind a number of the OSTI federated search applications. ]

ShareThis

Semantic processing is taking a big step forward.

From Mashable:

Google Acquires Metaweb to Improve Search

Google has acquired semantic web and real world database company Metaweb, a move the company says will help them ?improve search and make the web richer and more meaningful for everyone.?

We wrote about Metaweb back in 2008 when they received a significant chunk of funding to the tune of $42 million, on top of their first round of $15 million back in 2006. Since then the company has built its Freebase open database into a collection of over 12 million items from entertainment (movies, books, TV shows) to locations, celebrities, companies and other ?real world? objects. Google says the plan is to preserve and further develop the database and hope to enlist other companies to make use of and contribute to the data.

In addition to fleshing out Freebase, Google also hopes to leverage Metaweb to enhance its efforts with features like rich snippets and search answers, both of which aim to give back ?smarter? and more immediate results to specific queries. Right now, simpler requests like ?Barack Obama birthday? and ?events in San Jose? can spawn relevant answers right at the top of the search results page, but Google hopes to take this initiative further by feeding in more facts about the real world from Metaweb?s data repository.

Resource Shelf has some very insightful thoughts on the acquisition.

Here’s a good video on what Metaweb is about:

Here’s Google’s announcement of the acquisition.

ShareThis

Hope wrote again about WorldWideScience.org. Her article is at her blog, Signifcant Science. Hope is a research information technologist for a health network in Oregon. She is also Web administrator of the free online grants and scholarship listing service, ScanGrants, and of the free online search platform, ResearchRaven. From several conversations with Hope I know that ScanGrants is a labor of love and a good demonstration of Hope’s passion about helping researchers.

In Multilingual WorldWideScience: Accelerating Scientific Research, Empowering Researchers Hope reminds us of the key role that search plays in research especially in the world of free science and foreign language science.

Hope’s message is personal, and I love that:

As someone who grew up in a family that housed students who had left home and family in China, Japan, Iran, Korea and other countries to study engineering, chemistry, physics, biochemistry and so on at Oregon State University here in my hometown of Corvallis, Oregon I know what brilliant people there are in many countries who have so much to offer and what a boon it will be that the work of researchers worldwide will become useable to each of them and benefit the rest of us.

This update on Hope’s friend who suffered from ALS is even more touching:

I have recently lost a friend to amyotrophic lateral sclerosis and I would often sadly reflect as I bicycled home from her house about the glacial pace of progress on research on that disease and others like it. That is why I find Dr. Warnick’s enthusiasm and practical accomplishments so very admirable and the best possible case for paying one?s taxes with a minimal amount of grumbling. He is putting federal funds to exemplary use

Dr. Warnick, Director of OSTI, conceived WorldWideScience and his agency hosts and manages the search portal.

Databases and search engines aren’t about getting one’s job done. At the noblest level, they’re about solving important problems, and saving lives when we can.

[ Disclaimer: OSTI is one of my consulting clients. Deep Web Technologies, who built the single and multiple language search engines behind WorldWideScience.org and who sponsors this blog is another of my clients. ]

ShareThis

On July 1, 2010, Google announced an agreement to acquire ITA Software, a Cambridge, Massachusetts flight information software company, for $700 million, subject to adjustments.

Google’s acquisition of ITA Software will create a new, easier way for users to find better flight information online, which should encourage more users to make their flight purchases online.

The acquisition will benefit passengers, airlines and online travel agencies by making it easier for users to comparison shop for flights and airfares and by driving more potential customers to airlines’ and online travel agencies’ websites. Google won’t be setting airfare prices and has no plans to sell airline tickets to consumers.

Because Google doesn’t currently compete against ITA Software, the deal will not change existing market shares. We are very excited about ITA Software’s QPX business, and we’re looking forward to working with current and future customers. Google will honor all existing agreements, and we’re also enthusiastic about adding new partners.

The deal is subject to regulatory approval.

Travel news site tnooz made an interesting comment on Google’s new “troogle” service:

A user goes to native search in a Google search box, for example, and types the following query:

    Lowest fare to NYC on Sept 12th back on Sept 15th 2 people.

Google may respond with a display that shows a results box and the user selects from the list and, Voila, the user is now deep inside the workflow of the airline’s booking engine.

No messy metasearch, no expensive online travel, no blood curdling GDS control.

The price is guaranteed to be the lowest, so no surprises for the customer who now has a trustworthy result.

Note the emphasis, mine, on “No messy metasearch.” That got my attention. While I’m not a super frequent flyer I fly often enough to be annoyed at how difficult it is to find the best combination of flights with the least total travel time, the fewest hops, the most convenience, and the lowest cost. If troogle can really eliminate this exercise in frustration and drop me “deep inside the workflow of the airline’s booking engine” at the lowest price then they’ll earn my business.

The take-away for me here is that less search can be better. Wasn’t it Roy Tennant who said that only librarians like to search, everyone else likes to find? Don’t you think it’ll be the case that when the computer knows exactly what you want, without your needing to provide it with detailed context for every search — i.e. when search becomes ubiquitous and invisible — that that’s when search will travel to new heights?

What do you think?

ShareThis