Federated Search

Systematic Literature Review and Federated Search

Sol — Fri, 27 Apr 2012 02:56:00 +0000

The International Journal of Software Engineering & Applications has published the article: “A Federated Search Approach to Facilitate Systematic Literature Review in Software Engineering.” Here’s the abstract:

To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. Nowadays, Systematic Literature Review (SLR) has become a prominent methodology in evidence-based researches. Although adopting SLR in software engineering does not go far in practice, it has been resulted in valuable researches and is going to be more common. However, digital libraries and scientific databases as the best research resources do not provide enough mechanism for SLRs especially in software engineering. On the other hand, any loss of data may change the SLR results and leads to research bias. Accordingly, the search process and evidence collection in SLR is a critical point. This paper provides some tips to enhance the SLR process. The main contribution of this work is presenting a federated search tool which provides an automatic integrated search mechanism in well known Software Engineering databases. Results of case study show that this approach not only reduces required time to do SLR and facilitate its search process, but also improves its reliability and results in the increasing trend to use SLRs.

The article makes a good case for automating the search process to minimize the chance of missing important information in a literature review. The authors’ work in building a customized federated search engine has had three positive results:

1- It considerably reduces required time as one of the most concerns in SLR. It also improves the search process by including synonyms which are provided by an expert domain, automating the search process rather than manually search in every database for every search criteria, and finally integrating multiple databases search results.

2- Its crawler-enabled feature, facilitate search process and automatically save results in a database. After doing some researches, this database will contain thousands of records which not only could be used locally, but also would be so beneficial as a knowledge base for ongoing researches.

3- It facilitates both the qualitative or quantitative analysis on search results while they are integrated in a database. For example, classifying results based on their meta-data fields e.g. authors, may help the researcher to identify duplicated papers.

All in all, a nice article on a nice twist to federated search.

Abe Lederman gets in the (mobile) groove

Sol — Mon, 16 Apr 2012 21:37:44 +0000

Abe Lederman, founder and CEO of blog sponsor Deep Web Technologies, recently got a couple of exposures at MobileGroove, a site which provides analysis and commentary on mobile search, mobile advertising, and social media. The two MobileGroove articles cover Deep Web Technologies’ Biznar mobile federated search app.

More at the Deep Web Technologies Blog.

Michael Schrenk on Webbots, Spiders, and Screen Scrapers

Sol — Mon, 02 Apr 2012 14:25:31 +0000

I produced this podcast because I was curious about intelligent web agents and noticed this new edition of Michael Schrenk’s Webbots, Spiders, and Screen Scrapers.

In this podcast, Michael Schrenk and I discuss webbots, spiders, and screen scrapers. These are the tools that allow developers to crawl the web, to mash up contents from multiple web-sites, to monitor sites for activity and to create intelligent agents to make purchases on their behalf. Of particular interest are the stories Mr. Schrenk shares of the intelligent webbots he has built.

Click to listen to or download podcast

Why read Webbots, Spiders and Screen Scrapers?

Gain a bottom-up understanding of what webbots are, how they’re developed, and things to watch out for.
Understand the mind set difference between traditional web development and webbot development
Learn how to get ideas for great webbot projects
Discover how PHP/CURL facilitates advanced file downloads, cookie management and more.
Reenforce what you learn with projects and example scripts
Learn how to leverage WebbotsSpidersScreenScraper_Libraries, the common set of libraries that the book uses to make writing webbots easy.
Learn from the author’s 11 year career of writing webbots and spiders.

About the author

Michael Schrenk has developed webbots for over 17 years, working just about everywhere from Silicon Valley to Moscow, for clients like the BBC, foreign governments, and many Fortune 500 companies. He’s a frequent Defcon speaker and lives in Las Vegas, Nevada.

How do the millennials search?

Sol — Thu, 29 Mar 2012 20:47:57 +0000

[ Note: This article was first published in the Deep Web Technologies Blog. ]

Here’s a paper worth reading: “A study of the information search behaviour of the millennial generation.” No, not because there are any earth-shattering conclusions, but you may want to read the article to confirm that what you already suspect to be true really is true. Here’s the introduction from the paper’s abstract:

Introduction. Members of the millennial generation (born after 1982) have come of age in a society infused with technology and information. It is unclear how they determine the validity of information gathered, or whether or not validity is even a concern. Previous information search models based on mediated searches with different age groups may not adequately describe the search behaviours of this generation.

Here’s the conclusion:

Conclusions. These findings indicate that the search behaviour of millennial generation searchers may be problematic. Existing search models are appropriate; it is the execution of the model by the searcher within the context of the search environment that is at issue.

Beyond telling us what we already know the paper gives insights as to how librarians can help students to become more sophisticated researchers. Areas in which librarians can add value include:

Verification of quality of Web information sources
A shift of focus from filtering content to first verifying its quality and then filtering
Developing an orderly methodology for performing research

The paper might provide insights that search engine developers could someday roll into their offerings targeted at students.

A new kind of search engine

Sol — Thu, 22 Mar 2012 01:53:12 +0000

[Editor's Note: I received this email from Azhar Jassal at sehrch.com. I like what he's up to so I thought I'd give him a plug by republishing his letter, with Azhar's permission.]

I wanted to make you aware of a new search engine that I have spent the last 15 months building: sehrch.com

This is a new breed of search engine, it is a “structured search” engine. This type of search engine queries both the document web and the semantic web harmoniously. I have developed a simple query language that allows a user to intertwine between both of these worlds.

The purpose of Sehrch.com is to complete a users overall information retrieval task in as short time as possible by providing the most informative entity centric result. This is accomplished by either accepting an unstructured query (just how mainstream search engines are used) and applying conceptual awareness or by making structured queries, something all current mainstream search engines are incapable of doing (as they only concern themselves with the document web/ not the semantic web), which in my opinion adds a whole new dimension to information retrieval systems.

Some examples:

(unstructured) – Madonna : http://sehrch.com/s/Madonna
top result: Madonna (the entertainer)
note: just like a mainstream search engine

(semi structured) – Justin (age<25) : http://sehrch.com/s/Justin-(age%3C25)
top result: Justin Bieber
note: here we are searching for the unfielded token “Justin” with the fielded token “age” less than 25. Sehrch.com recognises age as a property due to the data and schemas its harvested from the Semantic Web. The combination of unfielded and fielded tokens makes this a semi structured query.

(structured) – (type:pop singers) (age<20) : http://sehrch.com/s/(type:pop-singers)-(age%3C20)
top result: Miley Cyrus
note: we have searched for entities that are pop singers and less than the age of 20. Sehrch.com gathers its entity awareness from the Semantic Web. The ranking of all entities is based on a popularity versus relevancy algorithm.

(structured) – (type:building) (floors>150) : http://sehrch.com/s/(type:building)-(floors%3E150)
top result: Burj Khalifa
note: search for buildings that have more than 150 floors. Again awareness is gathered from the Semantic Web.

There are more examples and on the about pages at: http://sehrch.com/about

If you have any questions please don’t hesitate to ask. Any mention you could make of Sehrch.com will be appreciated as I’m thirsty for feedback.

Some notes on my background in case you were interested: I am a computer scientist who focuses mostly on big data problems. I have spent all of my spare time (and money!) on sehrch.com for about 15 months now. My linkedin: http://uk.linkedin.com/in/azjassal

Carl Grant on librarians as more than pipelines

Sol — Sat, 10 Mar 2012 17:53:42 +0000

Carl Grant recently published an article, Are librarians choosing to disappear from the information & knowledge delivery process?, at the CARE Affiliates Blog. It reads in part:

As librarians, we frequently strive to connect users to information as seamlessly as possible. A group of librarians said to me recently: “As librarian intermediation becomes less visible to our users/members, it seems less likely it is that our work will be recognized. How do we keep from becoming victims of our own success?”

This is certainly not an uncommon question or concern. As our library collections have become virtual and as we increasingly stop housing the collections we offer, there is a tendency to see us as intermediaries serving as little more than pipelines to our members. We have to think about where we’re adding value to that information so that when delivered to the user/member that value is recognized. Then we need to make that value part of our brand. Otherwise, as stated by this concern, librarians become invisible and that seems to be an almost assured way to make sure our funding does the same. As evidenced by this recently updated chart on the Association of Research Libraries website, this seems to be the track we are on currently:

The chart is not pretty if you’re a librarian trying to justify your existence. But, on a positive note, after you’ve gotten past the depressing chart Carl Grant lists seven suggestions for products the library world should be providing to patrons.

I recommend this article as a sobering read with a positive spin.

The Charleston Advisor gives Deep Web Technologies high marks

Sol — Sat, 03 Mar 2012 15:29:58 +0000

[ This article was originally published in the Deep Web Technologies Blog. ]

The highly regarded Charleston Advisor, known for its “Critical reviews of Web products for Information Professionals,” has given Deep Web Technologies 4 3/8 of 5 possible stars for its Explorit federated search product. The individual scores forming the composite were:

Content: 4 1/2 stars
User Interface/Searchability: 4 1/2 stars
Pricing: 4 1/2 stars
Contract Options: 4 stars

The scores were assigned by two reviewers who played a key role in bringing Explorit to Stanford University:

Grace Baysinger, Head Librarian and Bibliographer at the Swain Chemistry and Chemical Engineering Library at Stanford University
Tom Cramer, Chief Technology Strategist at Stanford University Libraries and Academic Information Resources

The review upon which the scores are based, is available at Stanford. (Click on the purl.stanford.edu link for access to the full text.) At just six pages, the review makes for a quick read. The first four pages describe the Explorit features, infrastructure and support, and makes the case for the partnership between Deep Web Technologies and Stanford University that led to the development of the locally branded xSearch federated search product. Pages five and six provide the reviewers’ critical evaluation of Explorit, references, and their bios.

Key points from the critical evaluation include:

“Compared to other federated search products, Stanford found that DWT offered the most compelling package of performance, features, and design.”
“While federated search engines’ performance is inherently limited by the performance of its target sites, DWT’s progressive delivery of results gives researchers near real-time response with the first set of results while the application assembles a complete set of hits from all sources.” More information about how near-real time response works is available at the Federated Search Blog.
Explorit was “the only service that included alerts, and the only service that allowed us to create customized “search engines” locally.
“DWT’s performance, good relevance ranking, and faceting capabilities are very helpful to users.”
“Because Abstracting and Indexing tools contain controlled vocabulary terms, when a user is searching xSearch, there are more discovery points than if they were searching Google Scholar or a publisher’s site.”

More observations are available in the review. More information about xSearch is available at Stanford. Our own press release about the review is available on our website. An Explorit overview is also available at our web-site.

Sebastian Hammer on federated search

Sol — Sun, 19 Feb 2012 05:02:36 +0000

The Harvard Library Innovation Laboratory at the Harvard Law School posted a link to a 23-minute podcast interview with Sebastian Hammer. Hammer is the president of Index Data, a company in the information retrieval space, including federated search.

Update 4/3/12: A transcript of the interview is here.

Hammer was interviewed about the challenges of federated search, which he addressed in a very balanced way. The gist of Hammer’s message is that, yes, there are challenges to the technology but they’re not insurmountable. And, without using the word “discovery service,” Hammer did a fine job of explaining that large indexes are an important component of a search solution but they’re not the entire solution, especially in organizations that have highly specialized sources they need access to.

I was delighted to hear Hammer mention the idea of “super nodes” to allow federated search to scale to thousands of sources. Blog sponsor Deep Web Technologies has used this idea, which they call hierarchical federated search for several years. Several of their applications search other applications which can, in turn, search other applications. In 2009, Deep Web Technologies founder and president Abe Lederman delivered a talk and presented a paper at SLA,
Science Research: Journey to Ten Thousand Source, detailing his company’s proven “divide-and-conquer” approach to federating federations of sources.

I was also happy to hear Hammer speak to the importance of hybrid solutions. Federation is appropriate for gaining access to some content and maintaining a local index works for other content. Neither alone is a complete solution. Deep Web Technologies figured this out some years ago. A good example of hybrid search technology is the E-print Network, a product of the U.S. Department of Energy’s Office of Scientific and Technical Information, (OSTI). Deep Web Technologies built the search technology, which combines information about millions of documents crawled from over 30,000 sites, with federated content. I have been involved with the crawl piece of the E-print Network for a number of years and can testify to the power of the right hybrid solution. In 2008 I wrote a three-part series of articles at OSTI’s blog explaining the technology behind the E-print Network. Part One is here.

In conclusion, I highly recommend the podcast for a good reminder that federated search isn’t dead and that it’s an important part of search.

Abe Lederman on discovery services

Sol — Tue, 14 Feb 2012 19:48:20 +0000

Deep Web Technologies president, founder, and CTO Abe Lederman shares some thoughts on discovery services at the Deep Web Technologies Blog.

A greater need than ever for multilingual federated search

Sol — Sun, 05 Feb 2012 15:44:14 +0000

Multilingual federated search, the ability to search and to view results from foreign language sources in your own language, may be just an interesting idea to some but there is a strategic value to the technology. Consider this article published by the BBC in March of 2011: China ‘to overtake US on science’ in two years. If the prediction of the UK’s national science academy, the Royal Society, proves true then sometime next year China will produce scientific research papers at a faster rate than the current leader, the U.S.

Researchers in the English-speaking world have mostly been restricted to searching only English language sources since the tools for simultaneously searching foreign language sources and for performing the translations haven’t existed until recently. Thus, opportunities to search scholarly journals in Chinese, Japanese, Portuguese and other languages associated with countries producing a great volume of science output are being missed. In an economic climate where performing research and getting products to market quickly translates to that competitive edge that leads to greater profits, being able to scour the research Web quickly, effectively, efficiently, and on an ongoing basis is critical to developing and maintaining a competitive edge.

Blog sponsor Deep Web Technologies has developed a patent pending multilingual search version of its Explorit federated search application that integrates the search and translation technologies making for a seamless and productive research environment for scientists, engineers, and researchers in business, science, and technology.

A publicly searchable deployment of Deep Web Technologies’ multilingual federated search is WorldWideScience.org. The site’s About page describes the application:

One-stop Searching of WorldWideScience Sources

WorldWideScience.org is a global science gateway comprised of national and international scientific databases and portals. WorldWideScience.org accelerates scientific discovery and progress by providing one-stop searching of databases from around the world (Architecture: What is under the Hood). Multilingual WorldWideScience.org provides real-time searching and translation of globally-dispersed multilingual scientific literature.

A recent article by search pundit Stephen Arnold at Beyond Search, Deep Web Technologies: Cracking Multilingual Search, provides a good overview of the history, challenge, and applications of the technology. Arnold is the creator of the Wizards series, in which he has interviewed dozens of leaders in the search industry, including Deep Web Technologies founder, president, and CTO Abe Lederman.

More information about Deep Web Technologies multilingual federated search technology is available in this blog’s archives on the subject. Articles include: