Showcase Year 1


  • Y. Shi, A. Karatzoglou, L. Baltrunas, M. Larson, and A. Hanjalic, “CARS2: Learning Context-aware Representations for Context-Aware Recommendations.” 2014. [PDF]

This work addresses two specific issues of context modeling: the consistency between latent features and interpretation of relevance between a user and a context (or an item and a context).

The approach consists of on a novel formulation of latent factor models based on context-aware representations of users and items. In this model, the context is defined in its own latent space, which can be independent of that of the user and item. In addition, instead of modeling the relevance between the user/item and context, the proposed approach includes an additional layer of latent space for the user given a context, and an additional layer of latent space for the item given a context. The latent factors representing the user/item in the new latent space are introduced as the context-aware representation of the user/item.

The context-aware representations can be interpreted much more clearly than the relevance between the user/item and the context. The context-aware user representation can be interpreted as the hidden properties (e.g., mood) that may influence the user under the context (e.g., Friday). Similarly, the context-aware item representation can be interpreted as the hidden properties (e.g., suitability for families) of the item under the context (e.g., Sunday afternoon).

This work addresses the Context aspect of the RICHeS paradigm as the model propose a novel method to exploit context in recommender systems.

  • A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” in Proceedings of the 8th ACM Conference on Recommender Systems, 2014. [PDF]

The paper presents a recommender system evaluation toolkit that allows complete control of the different evaluation dimensions taken into account in any experimental evaluation of a recommender system: data splitting, definition of evaluation strategies, and computation of evaluation metrics. Rival is an open source Java toolkit that can be used to evaluate results coming out of any recommendation framework such as Apache Mahout, MyMediaLite, and Lenskit. Rival is used to make sure that the results are comparable and reproducible. Rival provides a benchmarking framework for evaluation of recommender algorithms, and supports CrowdRec’s commitment to benchmarking and evaluation.


  • T. Brodt and F. Hopfgartner, “Shedding Light on a Living Lab: The CLEF NEWSREEL Open Recommendation Platform,” in IIiX’14: Proceedings of Information Interaction in Context Conference, Regensburg, Germany, 2014, pp. 223-226. [PDF]

The work presents the Open Recommendation Platform (ORP). ORP allows researchers to evaluate their news recommendation strategies with feedback from actual users. This induces an evaluation environment called “living lab”. Unlike traditional data-driven evaluation protocols, ORP reflects recommendations’ actual utility in terms of click-through-rates. The work describes ORP’s architecture, message schemes, data formats, and user interface. ORP has been used as integral part of CLEF NewsREEL to support news recommender evaluation. It represents a unique tool bridging academic and industrial research. Thereby, the work contributes to the evaluation of context-aware and real-time recommender systems both of which RICHeS reflects.

  • A. Lommatzsch and S. Albayrak, “Real-Time News Recommendation Using Context-Aware Ensembles,” in Advances in Information Retrieval, Springer International Publishing, 2014, vol. 8416, pp. 51-62. [PDF]

The computation of good recommendations for streamed data is a challenge due to the continuously change in the set of users and items. In addition, the relevance of items highly depends on the specific context and current trends. In the paper we develop and evaluate several different algorithms tailored to the scenario of stream-based news recommendations. We show that the performance of the recommender algorithms highly depends on the context. Based on this result, an ensemble is created learning a delegation strategy which recommender algorithm is best-suited for requests taking into account the request properties and the context. This research is highly related to the RICHeS paradigm used in the CrowdRec project since the the research focuses on social interaction data and the context-aware adaptation of recommender strategies.

  • R. Turrin, R. Pagano, P. Cremonesi, and A. Condorelli, “Time-based TV programs prediction,” in 1st Workshop on Recommender Systems for Television and Online Video at ACM RecSys 2014, 2014. 

Recommending in the settings of linear TV has several peculiarities that makes it different from standard video-on-demand domains. Among the others, (i) items are available only for limited and fixed time frames (depending on the EPG schedules), (ii) users typically watch items using a television and a remote control (with limited interaction) and providing only implicit feedback, (iii) users are used to a very responsive navigation, and (iv) there is a strong affiliation between a user and a channel (v) with respect to a certain temporal pattern.

For such reasons, recommending linear TV programs faces with several RICHeS challenges, in particular: the context (i.e., the time), the interaction (i.e., the implicit feedback represented by the watched programs), and the real-time (i.e., the requirement to serve recommendation within a strict time).

In this work we take into consideration the time as main user context and we profile the user for each time slot of the week. The user profile is built on the basis of the metadata related to the items consumed by the user. Furthermore, to attenuate the problem that time, a continuous variable, is actually treated as a discrete one, we blended “close” time slots (e.g., Monday from 8pm to 9pm is close to Monday from 9pm to 10pm and to Tuesday from 8pm to 9pm).

The recommendation process is managed as a continuous stream of data that feed and adapt the time-based user profiles and are used at real-time to generating recommendations.

In the experiments we considered a private, linear TV data set and we identified the most relevant features to recommend the user behaviour on television as (i) the channel a TV show or a movie is broadcast and (ii) its category (namely, the genre and the subgenre).

  • J J. S. Pedro and A. Karatzoglou, “Question Recommendation for Collaborative Question Answering Systems with RankSLDA,” in 8th international Conference on Recommender Systems (RecSys ’14), 2014. [PDF]

The paper proposes an algorithm called RankSLDA for modeling expertise in online collaborative question answering (CQA) communities. The authors aim to reduce waiting times for responding to questions by warning users about the presence of the questions that match their interest and expertise. They try to match experts to questions based on the experts’ given answers on similar questions. The authors see this matching problem as a recommendation task. Their proposed algorithm (RankSLDA) extends the supervised Latent Dirichlet Allocation model following the learning-to-rank paradigm. The authors use data from the Cross Validate community, part of the Stack Exchange network, for their experiment. The results show that the proposed algorithm outperforms the alternative approaches.

The paper is relevant to the CrowdRec project in two respects. First, it leverages crowd engagement since the algorithm is directly at encouraging users to participate more actively in online communities by making recommendations. Second, the paper also follows the RICHeS paradaigm on incorporating the user context metadata in the process of making recommendations.

  • B. Loni and A. Said, “WrapRec: An Easy Extension of Recommender System Libraries,” in Proceedings of the 8th ACM Conference on Recommender Systems, 2014. [PDF]

WrapRec ( is an easy-to-extend open source toolkit written with C# which contains high level services that can low-level algorithms. The main goals of WrapRec are to provide a flexible I/O, evaluation mechanism and code reusability. WrapRec provides a rich data model which makes it easy to implement algorithms for different recommender system problems, such as context-aware and cross-domain recommendation. The toolkit is written in C# and the source code is publicly available on Github under the GPL license. This product address three challenges of the RICHeS paradigm. First it addresses the Heterogeneity aspect since multiple heterogeneous sources can be exploited by this toolkit and secondly it addresses the Context aspect since the services provided in this toolkit can easily incorporate context for the recommender system algorithms. This product also addresses the Social aspect since the social information can also be exploited by the recommender system algorithms that are wrapped in this toolkit.

  • Y. Shi, M. Larson, and A. Hanjalic, “Collaborative Filtering beyond the User-item Matrix: A survey of State of the Art and Future Challenges,” ACM Computing Surveys, 2014. [PDF]

The paper presents a comprehensive review of more than 200 key references on recommender system research with the aim of providing recommendations scenarios exploiting new information beyond the conventional user-item matrix. The authors introduced the new information into two categories: rich side information with respect to users and items, and interaction information extracted from the interplay of users and items. Based on this extensive review, the authors discuss the main challenges that recommender systems technology faces when extending the existing techniques with new information on users and items. The focus on the paper is related to CrowdRec’s RICHes paradigm, which expresses the need for integration of user interaction data, context metadata, and social information of users in the process of making recommendations.

  • A. Said and A. Bellogín, “Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks,” in Proceedings of the 8th ACM Conference on Recommender Systems, 2014. [PDF]

This paper discusses the issue of reproducibility and fair comparison when evaluating recommender algorithms. Comparisons of recommender algorithms are often conducted in terms of prediction accuracy, and better evaluation scores are taken to indicate a better-performing recommender. However, due to differences in design and implementation of evaluation strategies, it is tricky to make a fair comparison of recommender algorithms. The paper compares the common recommender algorithms implemented in three popular recommendation frameworks: Apache Mahout, MyMediaLite, and Lenskit; using three datasets: MovieLens 100k and 1M, and Yelp. Such comparison has been done under a controlled evaluation protocol using the same dataset, data splitting, evaluation strategies and metrics. The results are ultimately compared with the results coming out of the internal evaluation of the three frameworks, which shows there are differences between these two evaluation methods. The paper provides guidelines on how to report evaluation results for the recommender algorithms in order to ensure reproducibility and comparison of the results. The paper is related to CrowdRec effort in the area of benchmarking and reference frameworks.

  • B. Hidasi and D. Tikk, “Approximate modeling of continuous context in factorization algorithms.,” in Workshop on Context-awareness in Retrieval and Recommendation, Amsterdam, 2014. 

Context information is assumed to be categorical in factorization algorithms, i.e. it is supposed to have discrete context-states in it and there is no ordering between these states. Some real life contexts however do not fall into this category. For example time – one of the most commonly used contexts – is continuous. Such dimensions are first transformed by discretizing them and ignoring the relations of these discrete context-states. This paper proposes two simple approximative modeling methods that integrates continuous context into factorization methods. The suggested methods are general and thus can be incorporated into any context-aware factorization method. We show its effectiveness by extending the iTALS algorithm and using seasonality as the context. The paper relates to the context related part of the project.

  • D. Loiacono, A. Lommatzsch, and R. Turrin, “An Analysis of the 2014 RecSys Challenge,” in Proceedings of the Recommender Systems Challenge 2014, at ACM RecSys 2014, Foster City, CA, USA, 2014. 

The prediction how news messages or tweets are perceived is a challenging task. In this paper we analyze movie related tweets and study strategies for predicting the user engagement. For this purpose we enrich crawled twitter messages with additional semantic data (retrieved from IMDb and Freebase). The task covers multiple RICHeS challenges, in particular focusing on the use of Heterogeneous and Social data (the collected tweet messages).

We implement different machine learning algorithms and feature selection strategies allowing us successfully predicting whether a twitter messages will be retweeted or not.

Our research shows that the context of a tweet as well as the social aspects (who tweeted) have a big influence on the number of retweets a tweet receives. The research is highly related to the CrowdRec objectives because the task covers the aspects social data, stream-based recommendations and context.

  • B. Kille, T. Brodt, T. Heintz, F. Hopfgartner, A. Lommatzsch, and J. Seiler, “Overview of CLEF NEWSREEL 2014: News Recommendations Evaluation Labs,” in CLEF 2014 Evaluation Labs and Workshop, Online Working Notes, Sheffield, UK, 2014, pp. 790-801. [PDF]

The work summarizes NewsREEL 2014 a competition dedicated to finding better strategies for recommending news. Participants could both work with recorded data and interact with users of an operating news recommender system. The latter represents a form of evaluation referred to as “living lab”. Participants contributed 42 recommendation methods competing with respect to click-through-rates (CTR). We observed that contextual factors affected the performance. Contextual factors include devices, weekday, and daytime. Participants experienced conditions similar to industrial recommender system. These conditions include restrictions such as response time limits and load peaks. The work contributes to research concerning real-time and context-aware recommender systems both relevant in the RICHeS paradigm.