Showcase Year 3

NewsREEL overview Paper

Benjamin Kille, Andreas Lommatzsch, Gebrekirstos Gebremeskel, Frank Hopfgartner, Martha Larson, Torben Brodt, Jonas Seiler, Davide Malagoli, Andras Sereny, and Arjen De Vries:
CLEF NewsREEL 2016: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms In Proceedings of the 7th International Conference of the CLEF Initiative, 5 – 8 September 2016, Evora – Portugal, LNCS vol. 9822, DOI: 10.1007/978-3-319-44564-9, Springer International Publishing [PDF]

Description: The realistic evaluation of recommender algorithms is a challenging task. In order to optimize recommender systems for real-live use cases, the evaluation must consider multiple aspects such as result precision and the technical complexity. In this paper we explain the challenges arising when evaluating stream-based recommender algorithms. We present an approach that support both online and offline evaluation of recommender algorithms. The approach has been applied in the CLEF NewsREEL challenge that offers a living lab task as well as an offline replay task. We discuss our experiences running the challenge. We explain the different perspectives on the evaluation focusing on the academic, precision-oriented view as well as on the technical complexity of the recommender systems. In addition, we discuss the perspective of the participants and the challenge organizers with respect to reproducibility, significance, and the robustness of the evaluation results.
This paper shows that the realistic evaluation of stream-based recommender algorithms in dynamic environments is a complex task. We show that the combination of multi-dimensional metrics and the evaluation of both online and offline evaluation results give deep insights in the strength and weaknesses of recommender algorithms. This enables the optimization of recommender algorithms with respect to real-live applications and business models.

Relation to CrowdRec: The paper explains the challenges when evaluating stream-based recommender algorithms in complex environments. The paper is strongly related to WP2 (“Reference Framework”) and WP3 (“Stream Recommendation Algorithms”). A realistic evaluation is the basis for the optimization of highly productive recommender algorithms. In the paper we discuss the evaluation in NewsREEL. The NewsREEL challenge explicitly considers heterogeneous, social interaction data in a real-time setting. These “RICHeS-challenges” (Real-time, Interaction, Heterogeneity, Social interaction) are addressed in the paper.  We show that multi-dimensional benchmarking is the key when development optimized algorithms for real-live applications.

Revolution Workshop PPF Paper

Tamas Motajcsek, Jean-Yves Le Moine, Martha Larson, Daniel Kohlsdorf, Andreas Lommatzsch, Domonkos Tikk, Omar Alonso, Paolo Cremonesi, Andrew Demetriou, Kristaps Dobrajs, Franca Garzotto, Ayse Göker, Frank Hopfgartner, Davide Malagoli, Thuy Ngoc Nguyen, Jasminko Novak, Francesco Ricci, Mario Scriminaci, Marko Tkalcic, Anna Zacchi: Algorithms Aside: Recommendation As The Lens Of LifeIn Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September 15-19, 2016: 215-219 [PDF]

Description: In this position paper, we take the experimental approach of putting algorithms aside, and reflect on what recommenders would be for people if they were not tied to technology. By looking at some of the shortcomings that current recommenders have fallen into and discussing their limitations from a human point of view, we ask the question: if freed from all limitations, what should, and what could, RecSys be? We then turn to the idea that life itself is the best recommender system, and that people themselves are the query. By looking at how life brings people in contact with options that suit their needs or match their preferences, we hope to shed further light on what current RecSys could be doing better. Finally, we look at the forms that RecSys could take in the future. By formulating our vision beyond the reach of usual considerations and current limitations, including business models, algorithms, data sets, and evaluation methodologies, we attempt to arrive at fresh conclusions that may inspire the next steps taken by the community of researchers working on RecSys.

Relation to CrowdRec: This paper represents the collective thoughts of a group of researchers thinking and discussing what the future of RecSys should be, if allowed to set aside the strengths and weaknesses of today’s algorithms. The paper is related to WP6 (“Exploitation and Dissemination”) and Task 6.3 (“Business modeling and exploitation”). In the paper we assess the current limitations and pitfalls of RecSys, ponder the notion of whether “life” could be a model for RecSys, and lastly, present our ideas about the kind of recommenders that could exist in the future.

Deep learning papers (ICLR16 and RecSys paper)

Session-based recommendations with deep learning

Hidasi, A. Karatzoglou, L. Baltrunas, D. Tikk: Session-based Recommendations with Recurrent Neural Networks. At International Conference on Learning Representations (ICLR 2016), 2-4 May, 2016, San Juan, Puerto Rico. [PDF]

Public code:

Related slides (1st Budapest Recsys and Personalization Meetup):

Description: The paper focuses on the frequent problem of session based modeling, which is seldom researched in recsys literature. Using the whole user history on many sites is not sensible, because either (a) the users can’t be identified properly; and/or (b) their behavior is session-based. This is the typical setting in most e-commerce sites (e.g. where usage is “need-based” rather than “preference-based”) and on large content platforms (e.g. if the content is very diverse). In such domains, users are in a perpetual state of cold-start, because their earlier visits are practically useless for predicting their interests in the current session. Therefore modeling the session on-the-fly is essential for good recommendations. We use a special form recurrent neural networks – the Gated Recurrent Unit (GRU) – to model sessions. However, using the off-the-shelf network is inconvenient for the recommendation task, due to the specialities of the problem. Therefore we propose three major modifications to adapt the algorithm to the task: (1) session-based mini-batching; (2) pairwise ranking objective function; (3) sampling the output of the network. The resulting algorithm achieves +15-25% in recommendation accuracy over the industry de-facto solution – item-kNN – for this problem, while maintaining practically acceptable training times.

Relation to CrowdRec: The paper addresses session-based recommendations, which is a very common setup in real life recommender systems. Within CrowdRec, it relates to the deep learning effort of the research consortium and is the first publication of the cooperation between GRA and TID on deep learning research. It also relates to the context-aware aspect of the project as in this scenario the previous interactions of the session are the context of the recommendation.

Adding content features to session-based recommendations (multimedia analysis)

Hidasi, M. Quadrana, A. Karatzoglou, D. Tikk: Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations. In Proceedings of ACM Conference on Recommender Systems (RecSys 2016), 15-19 September, 2016, Boston, USA. [PDF]


Description: This work is the continuation of the deep learning research on session-based recommendation. User clicks on items mainly depend on the information available to them, which include their apriori knowledge about items, as well as the information presented on the site. The latter mainly consists of text (e.g. title or description) and image (e.g. product image, video thumbnail). Our hypothesis is that including this information results in better session models. As it turns out, just adding this information to the input of the RNN is not enough to leverage its full potential. Therefore we propose the parallel RNN (p-RNN) architecture, where different information sources are processed by separate RNNs, which are then connected before producing the output. This special architecture also requires special training methods. Inspired by alternating least squares and similar methods, we propose three alternating training strategies that help p-RNNs achieve peak performance. Ranking is improved by 5-15% over the RNN that uses item IDs only as its input and we also show that the additional information sources can improve accuracy even if increasing the capacity of the network doesn’t help.

Relation to CrowdRec: This paper introduces a solution to efficiently integrate core content features, such as image and text data into session-based recommendations. It relates to the multimedia and context aspects of CrowdRec.

RecSys Challenge Overview

Fabian Abel, András Benczúr, Daniel Kohlsdorf, Martha Larson, and Róbert Pálovics. 2016. RecSys Challenge 2016: Job Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM, New York, NY, USA, 425-426.

Related slides:

Recommendation Challenges on XING. 6th RecSys NL meetup, September 2016, Amsterdam:

RecSys challenge workshop, September 2016, Boston:

What’s wrong with Recruiter-John? – A nontrivial recommender challenge. 2nd RecSys Meetup, June 2016, Budapest:

Get on with it! RecSys industry challenges move towards real-world online evaluation. ECIR 2016, March 2016, Padova:

Description: This paper provides an overview of the 2016 RecSys Challenge on job recommendation using data and a use case from XING. The challenge had the following formulation: given a XING user, the recommender algorithms had to predict those job postings with which a user will interact (e.g., click or bookmark). In total, 366 teams registered to participate in the challenge, of which 119 were active participants. In total, 4232 solutions were submitted.

The dataset contained approximately 1.4 million distinct users (job seekers) and around the same number of distinct items (job postings). The algorithms of challenge participants were expected to exploit the multiple sources of information that were included in the data, including codes representing information such as skills, education level, experience level, employment type, and location for users and for job postings. The dataset was a semi-synthetic sample of historic XING data, i.e., it was a sample of XING data enriched with noise for the purpose of anonymization and to abstract from real user profile data.

A major challenge that the participants faced was addressing the extreme sparseness of the data: for 24% of the items and 43% of the users, the training data did not include even a single interaction. This level of sparseness is typical for the real-world use case of job recommendation at XING. The teams that were successful in the challenge were teams who consciously focused on mitigating the effects of sparseness. Standard matrix factorization approaches were not appropriate given these data conditions. The challenge was dominated by simple methods that were thoughtfully designed and combined. Industry teams in general outperformed academic teams, revealing the gap between the types of problems that these two research worlds consider standard. The challenge could be addressed effectively without computationally complex models or extensive computational resources.

Relation to CrowdRec: The RecSys challenge requires the combination of multiple sources of information under real-world conditions, addressing CrowdRec Objective 1 “Stream Recommendation”. The data set anonymization method developed for the challenge addresses a goal of CrowdRec WP2 in the area of data set development. The challenge helped to narrow the gap between the research carried out in academic settings and real-world recommender systems projects. As such, it contributed to the goals of reaching the larger recommender system research community, fulfilling an important goal of WP6, which is dedicated to dissemination and exploitation.

Music as Technology

Demetriou, A., Larson, M. and Liem, C.C., (2016). Go with the flow: when listeners use music as technology. In Proceedings of ISMIR 2016. [PDF]

Related Slides:


Description: Music has been shown to have a profound effect on listeners’ internal states as evidenced by neuroscience research. Listeners report selecting and listening to music with specific intent, thereby using music as a tool to achieve desired psychological effects within a given context. In light of these observations, we argue that music information retrieval and recommendation research must revisit the dominant assumption that listening to music is only an end unto itself. Instead, researchers should embrace the idea that music is also a technology used by listeners to achieve a specific desired internal state, given a particular set of circumstances and a desired goal. This paper focuses on listening to music in isolation (i.e., when the user listens to music by themselves with headphones) and surveys research from the fields of social psychology and neuroscience to build a case for a new line of research in music information retrieval on the ability of music to produce flow states in listeners. We argue that interdisciplinary collaboration is necessary in order to develop the understanding and techniques necessary to allow listeners to exploit the full potential of music as psychological technology.

Relation to CrowdRec: The Kollekt business model has led the company to be interested in the question of how music impacts the brain during activities. This paper describes the neuroscience foundations for the role of music during activities that require focus. It is related to the work done on supporting Kollekt curators in CrowdRec WP5. It is also related to the signal processing techniques developed in WP4 for automatically pre-processing music in order to allow humans to make more useful contributions to recommendation.


Context-driven Past-Present-Future Paper

Roberto Pagano, Paolo Cremonesi, Martha Larson, Balázs Hidasi, Domonkos Tikk, Alexandros Karatzoglou, and Massimo Quadrana. 2016. The Contextual Turn: from Context-Aware to Context-Driven Recommender Systems. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM, New York, NY, USA, 249-252. [PDF]

Description: With this paper, CrowdRec studied current developments in industries in which recommender systems make a key contribution to business models. We also looked at evolving algorithmic trends. What emerges is a picture of an ongoing “Contextual Turn”. In other words, we found that today’s recommender system use scenarios often must use context as a primary, rather than a secondary, source of information. Context is understood in the classic sense of time and place, but we also extend our notion of context to include interactions in a user session. The “Contextual Turn” means that more sources of information than ever before must be combined, and that algorithms must be able to handle large amounts of data. The work on deep learning in the project addresses these challenges. The paper finishes with a section that sets out potential benefits of the “Contextual Turn” above and beyond improved recommendations from the user point of view. These benefits are: Popping the filter bubble (exposing users to variety), leveling the playing field (eliminating unintentional bias, e.g., as related to gender), protecting privacy (using interaction data that is not associated with a user ID), and adding transparency (providing explanations that are based on situations and user goals).

Relation to CrowdRec: During the CrowdRec requirements process (WP2), it became clear that there was a pressing need for session-driven recommendation, i.e., algorithms capable of making predictions on the basis of user interactions during a session. The project became interested in using session interaction as a “proxy” for context. At the same time, we became aware of the large number of cases in which user ID information is not reliable or not available at all. Several algorithms were developed in WP3, the “Stream Recommendation” workpackage. For there, we realized that research in the recommender system community on context-driven recommendation was fragmented: different research efforts were using context as a primary source of information for prediction, but were largely not building on each other’s insights. To support the bundling of efforts that could make this research more effective in the future, we examined the algorithms and the business models in the context of our work on exploitation in WP6 and wrote this paper.