I'm currently implementing a recommender system using implicit ratings (time spent on an article) but I'm wondering what would be the proper metric to assess the system. MSE doesn't seems to fit to this usecase.

My algorithm is based on the paper : Collaborative Filtering for Implicit Feedback Datasets


1 Answer 1


Evaluating implicit feedback based recommendations is tricky. Here are couple of approaches that I'd recommend.

Approach 1

You can use modified precision and recall metrics. Overall the procedure would be as follows,

  1. Divide your data into train and test set by users.
  2. For a user in test set, given their history, get the top N recommendations using implicit feedback based model.
  3. Precision can be calculated using # of recommendations given by model which actually matched by what user had acted upon (for example read in case of articles).
  4. Recall can be calculated using # of user actions (articles read by user) that were captured by top N recommendations.
  5. You can calculate these for all users in test set and average them.

Approach 2

The approach is similar to approach 1, but rather than splitting train and test data by users, you use something called as "leave one out" strategy. We use a simple accuracy metric in this case. The steps to calculate the accuracy will be as follows,

  1. For each user (or a subset of users), hide one of the articles read/browsed and move it to test set (leave one out).
  2. Using user history, get top N recommendations for each user.
  3. Calculate the number of times the left out article was captured by the top N recommendations.

Note that in both cases, the N in top N recommendations will become your hyper-parameter which can be tuned further.

  • $\begingroup$ Thank you very much ! For the first approach, how can I handle the test users ? I need to get their latent representation in order to get the recommendations. Do I need to compute it through the "fold-in" technique ? $\endgroup$ Commented Aug 2, 2016 at 9:57
  • $\begingroup$ @user3195078 I am not aware of what fold-in technique is. $\endgroup$
    – hssay
    Commented Aug 3, 2016 at 4:00
  • $\begingroup$ The thing is, if I don't use the "test users" for the learning step, how do I get their "latent" representations in order to compute their recommendations ? $\endgroup$ Commented Aug 4, 2016 at 15:31
  • $\begingroup$ ^ The second option I've highlighted above should be useful in the case you mentioned (getting latent representations for recommendations). The learning step considers all users, but hides some of their engagement history/implicit preferences. $\endgroup$
    – hssay
    Commented Aug 8, 2016 at 4:34
  • $\begingroup$ Yes the second is perfectly fine, I think I'll go for it, thanks again :) $\endgroup$ Commented Aug 10, 2016 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.