There are three main classes of comparison coming to mind in our context:

1. Search – comparing patterns and frequencies within and across items. Example, text query against documents.
2. Recommendation – comparing items based on their occurrence. Example, collaborative filtering of co-occurring items
3. Classification/clustering – comparing items based on their features. Example, clustering and merging (near) duplicate items.

There are products along these classes of comparison available, e.g. Lucene or Solr. The problem is that products specialise on a use case, for example search in case of Lucene and Solr.  The specialisation commonly focuses on one or small subset of aspects of the information we have available, e.g. patterns or relationships. Some of the data are only poorly or not at all utilised and comparison across types is often impossible or hard. Moreover, we have to (internally) in many situations do similar comparisons but utilizing specialised products is not always a sensible approach. Where we do use existing technologies and algorithms we are limited by their abilities and insight (or lack of).

We pose these challenges:

• To unify the data comparison classes in one system to extract value from the full data set (patterns, frequencies, relationships, co-occurrence, …) and access it transparently from different services (search, recommendation, de-duplication) according to their needs.
• To scale it for Mendeley (to 10^8 and beyond).
• To be as effective or even better than dedicated products.

We will solve these challenges as part of the TEAM project applying and extending state-of-the-art research. The outcome will a) extend knowledge in form of peer reviewed research publications, and b) result in a real-world, working system at Mendeley.