Success stories

These are our CrowdRec stories about the success of research within the project. We describe for each “story” what the state of the art was before the project and what the project contributed, as well as what the state of the art was after the project, from the point of view of the end user of the recommender system.


Deep Learning (Gravity)

Depending on the products/content available on a website, the usefulness of the long term user histories of visitors for use in recommendations also changes. In the broad category of “taste based” domains, such as books, music, movies, etc. past behavior can be used (most of the times) to guess what the users are looking for and recommend suitable items. In “need based” domains, such as electronics, general e-commerce, etc., the recommender should suggest items that the users need at that moment and past behavior is mostly irrelevant for determining this. For example, if someone wants to buy a new fridge, the information on which washing machine she bought 2 years ago has no real relevance. The behavior in this case is session based: it is specific only for the current browsing session. This type of behavior is also common on content sites (e.g. online video portals) as users a looking to immerse themselves in various topics. It is also important in cases where users don’t need to log in to use a service or hard to identify for other reasons. In these scenarios, recommenders must work without any prior knowledge, but recommend relevant items as soon as the first interaction of the user with the site. Most of the online services require session-based recommendations.

Recommenders were very simplistic for this scenario before CrowdRec. The usual solution was to recommend items similar to the item that the user is currently interacting with. The weakness of this approach is that it doesn’t consider previous interactions of the user and thus doesn’t get more accurate as the user interacts more and is more likely to recommend irrelevant items. We used deep learning – Gated Recurrent units to be more precise, which is a form of Recurrent Neural Networks (RNN) – to model sessions based on the users’ interactions with items. We designed a new neural network (GRU4Rec) in order to suit the recommendation problem and comply with strict technical requirements on training and recommendation times. This solution produces significantly more accurate recommendations than the item-to-item approach.

We even took this approach one step further and included item features that the user sees on the screen when being recommended to; for example video thumbnail,  product images, titles, short descriptions. This information is one of the key components on which the users base their decision of clicking on the recommendation. To include this information, we designed a novel neural network architecture – the parallel RNN (p-RNN) – and an appropriate training procedure. This further increased recommendation accuracy.


Music for daily activities (Kollekt)

People have always sought a connection between music and their mood. However, nowadays, we see an increasing number of people who want music to match their activity. For certain activities (for example, working out at the gym) it is relatively easy to find the necessary music. However, for other activities, such as getting work done, or having business discussions, very little is known about what kind of music is appropriate. CrowdRec collected evidence from research carried out in the areas of neuroscience and psychology that has a bearing on the use of music for life activities. Their findings point to the importance of several signal level properties of music, including tempo, lyrics, and acoustic distractors. Currently, recommender systems do not take these properties into account when recommending music. As a result of research with the CrowdRec project [Demetriou, A., Larson, M. and Liem, C.C., (2016). Go with the flow: when listeners use music as technology. In Proceedings of ISMIR 2016.], Kollekt curators have access to information about signal level properties that are important for finding music that provides an appropriate background for people’s daily activities. The signal analysis tools developed by CrowdRec within the CurRec CCP can support them by speeding up the process of filtering music with the right properties, allowing the curators to create new playlists more quickly and with confidence.


Feedback App (XING)

Before XING joined the CrowdRec project, there were no feedback mechanisms in place that allowed users of the XING recommender systems to provide explicit feedback. The recommender algorithms were solely relying on implicit feedback such as clicks in order to understand whether a user perceived a recommendation as relevant or not. As part of the CrowdRec requirement analysis, we identified that “providing engaging feedback mechanisms” was one of the top four challenges that we aimed to solve during the CrowdRec project.

Therefore, CrowdRec partners designed and implemented the first version of the so-called CrowdRec Feedback App which was deployed and released to the XING users just two months after the requirement analysis. In its first version, it allowed XING to ask their users survey-like questions (e.g. “How happy are you with your job recommendations in general?”) and enabled XING users to rate the relevance of their recommendations on a five-star rating scale. During the project, CrowdRec partners developed the Feedback App towards a more powerful feedback tool with lightweight gamification elements and playful functionality for explaining why an item was recommended to the user (c.f. “Is this job for me?” feature).

Today, the Feedback App is used for all main recommender systems on XING. The CrowdRec Feedback App increased user engagement and allowed XING’s Data Science team to follow a new modus operandi when it comes to the development of their recommender systems (e.g. new KPIs such as “user satisfaction scores” and written feedback allow the team to quickly react to trending issues). Finally, the feedback and particularly the more than 800,000 ratings that were collected during the CrowdRec project, led to various new models and algorithms that were implemented and deployed on XING. Those feedback-based algorithms increased the relevance of XING’s recommendations significantly (e.g. click-through rates of XING’ job recommendation system more than doubled over the course of the CrowdRec project). CrowdRec models for predicting relevance ratings moreover allowed for new applications such as identifying best matching candidates which is today the core of a new XING product: the XING referral manager (