Here’s a quick example on how to use it:

Get the top user and his expertise score:

Get the top resource and its quality score:

You can also use the library to simulate the HITS algorithm of Jon Kleinberg. Simply supply a credit score function C(x) = 1 to the SPEAR algorithm (see the documentation of the Spear.run() method).

Feel free to play around with it and send me feedback!

PS: The SPEAR Python library requires SciPy/NumPy. If you don’t have these installed already, here are some installation instructures to get you started.

Thanks again to Vik Singh and Ariel Seidman from Yahoo! for this great opportunity and for their support!

The TR article A Better Way to Rank Expertise Online summarizes our approach quite well, and we’re really happy about the very positive feedback therein from researchers like Professor Jon Kleinberg (the inventor of the HITS algorithm on which our SPEAR algorithm is based), Scott Golder (his publication together with Huberman on usage patterns in collaborative tagging systems is one of the earliest and currently most cited works in the area of the Social Web) and Ciro Cattuto (very nice work on social dynamics and network characteristics of folksonomies). Thanks!

Abstract

With a suitable algorithm for ranking the expertise of a user in a collaborative tagging system, we will be able to identify experts and discover useful and relevant resources through them. We propose that the level of expertise of a user with respect to a particular topic is mainly determined by two factors. Firstly, an expert should possess a high quality collection of resources, while the quality of a Web resource depends on the expertise of the users who have assigned tags to it. Secondly, an expert should be one who tends to identify interesting or useful resources before other users do.

We propose a graph-based algorithm, SPEAR (SPamming-resistant Expertise Analysis and Ranking), which implements these ideas for ranking users in a folksonomy. We evaluate our method with experiments on data sets collected from Delicious.com comprising over 71,000 Web documents, 0.5 million users and 2 million shared bookmarks. We also show that the algorithm is more resistant to spammers than other methods such as the original HITS algorithm and simple statistical measures.

Full Paper & Presentation

• M. G. Noll, C.-M. Au Yeung, N. Gibbins, C. Meinel, N. Shadbolt
  Telling Experts from Spammers: Expertise Ranking in Folksonomies
  Proceedings of 32nd ACM SIGIR Conference, Boston, USA, July 2009, pp. 612-619, ISBN 978-1-60558-483-6 (ACM Link, BibTeX)
• Presentation: Telling Experts from Spammers (Talk), our talk at SIGIR 2009

Related Links

• The Spear Algorithm – website dedicated to SPEAR
• List of my publications
• Article on Technology Review about this work: A Better Way to Rank Expertise Online, July 2009
• 32nd Annual ACM SIGIR Conference, Boston, USA, July 2009

Here’s a brief teaser:

Writing a Personal Link Recommendation Engine

There is so much going on in the Internet today that it’s hard to keep track of it all. Whether you need to stay up to date with job-related information, the current state of research, or the latest developments in the Open Source world, finding relevant information quickly on the Internet is difficult. So-called “social” services such as Delicious.com or Digg.com try to support users by allowing them to collaboratively share their knowledge about interesting web sites. In this article, we will design and implement a simple, yet powerful link recommender whose analysis is based solely on Delicious.com social bookmarks information.

Interested? Head over to PyMag and have a look!

CABS120k08 is a large research data set about Web metadata based on a sample of 120,000 web documents in 2008 (=120k08) with data retrieved from the Open Directory Project, the AOL Search query log corpus AOL500k, Google PageRank, Delicious.com, and anchor text from incoming hyperlinks.

The data corpus is described in detail in my paper The Metadata Triumvirate: Social Annotations, Anchor Texts and Search Queries, for which the corpus was built. Enjoy!

Abstract

In this case study, we describe the design and architecture of a scalable collaborative web filtering service, TaggyBear, which is powered by free and open source software. We will introduce the reader to the ideas and concepts behind TaggyBear, and discuss why we picked the software components that form the basis of the service. We will talk about how we combined or extended their functionality to build the TaggyBear service, and provide some initial benchmarking results and performance figures.

Note for fellow developers: The described system components include Pylons, MySQL, Tokyo Cabinet/Tokyo Tyrant, Pylog (custom app based on Twisted), Hadoop, and on the client side a Firefox Add-On.

Full Paper & Presentation

• M. G. Noll, C. Meinel
  Building a Scalable Collaborative Web Filter with Free and Open Source Software (PDF)
  Proceedings of 4th IEEE Signal-Image Technology & Internet-based Systems (SITIS), IEEE CS Press, Bali, Indonesia, November 2008, pp. 563-571, ISBN 978-0-7695-3493-0 (IEEE Link, BibTeX)

Related Links

• List of my publications
• Design and Anatomy of a Social Web Filtering Service, Proceedings of 4th International Conference on Cooperative Internet Computing (CIC), Hong Kong, October 2006, pp. 35-44, ISBN 978-981-281-109-7
• 4th IEEE International Conference on Signal-Image Technology & Internet-based Systems (SITIS), Bali, Indonesia, November 2008