Evaluating recommender systems with (implicit) binary ratings only

Question

I'm analyzing a set of news articles and user libraries. User library is the set of news articles shared by one user. Obviously, the rating is 1 (the article is in user's library) and 0, otherwise. I hypothetically present every user with M articles sorted by their predicted rating and evaluate based on which of these articles were actually in the library. So, my only evaluation metric is

recall@M = # of articles the user shared in top M / total # of articles the user shared

Using precision doesn't make sense as zero ratings are uncertain (a user is not interested in the article or is just unaware of it). If I can't afford the user studies (explicit user feedback on recommendations), what are other evaluation metrics can I use?

Answer 1

I suggest to use Expected Utility or R-score. Assume your model has created an ordered list of recommendations where the first item is the one the user is most likely and the last is the one the user is least likely interested in. Let's say that this recommendations are specified by $r_i$ where $i$ is the position in the list.

Expected utility for a particular user u is now defined as: $R_u=\sum_{i=1}^{n}\frac{f_u(r_i)}{2^{\frac{i-1}{\alpha-1}}}$

where ...

• $f_u(r_i)=1$, if the recommended item $r_i$ is the user's library, else 0
• $\alpha$ specifies the slope of the decay. The smaller alpha, the greater the slope.

This metric measures how likely it is that a user will view a recommended item, assuming a list of items is recommended to him. If the item of interest is placed at a very high position, it is unlikely that the user will bother to scroll/look that far, even if it exactly what he/she wanted. In this sense, $\alpha$ specifies how patient the average user is.

To calculate the R-score for a set of users, it is recommended to normalize the R-score per user beforehand. The resulting score is

$R =\sum_u\frac{R_u}{R_u^*}$

where $R_u^*$ is the maximum R-score you can achieve for one user. That is, given a user has k items in his library you want to predict, the items at the first k positions are exactly these ones.

For more metrics or as a general read I recommend Evaluating Recommendations Systems by Shani & Gunarwadana