I'm currently implementing a recommender system using implicit ratings (time spent on an article) but I'm wondering what would be the proper metric to assess the system. MSE doesn't seems to fit to this usecase.

My algorithm is based on the paper : Collaborative Filtering for Implicit Feedback Datasets


Evaluating implicit feedback based recommendations is tricky. Here are couple of approaches that I'd recommend.

Approach 1

You can use modified precision and recall metrics. Overall the procedure would be as follows,

  1. Divide your data into train and test set by users.
  2. For a user in test set, given their history, get the top N recommendations using implicit feedback based model.
  3. Precision can be calculated using # of recommendations given by model which actually matched by what user had acted upon (for example read in case of articles).
  4. Recall can be calculated using # of user actions (articles read by user) that were captured by top N recommendations.
  5. You can calculate these for all users in test set and average them.

Approach 2

The approach is similar to approach 1, but rather than splitting train and test data by users, you use something called as "leave one out" strategy. We use a simple accuracy metric in this case. The steps to calculate the accuracy will be as follows,

  1. For each user (or a subset of users), hide one of the articles read/browsed and move it to test set (leave one out).
  2. Using user history, get top N recommendations for each user.
  3. Calculate the number of times the left out article was captured by the top N recommendations.

Note that in both cases, the N in top N recommendations will become your hyper-parameter which can be tuned further.

  • $\begingroup$ Thank you very much ! For the first approach, how can I handle the test users ? I need to get their latent representation in order to get the recommendations. Do I need to compute it through the "fold-in" technique ? $\endgroup$ – user3195078 Aug 2 '16 at 9:57
  • $\begingroup$ @user3195078 I am not aware of what fold-in technique is. $\endgroup$ – hssay Aug 3 '16 at 4:00
  • $\begingroup$ The thing is, if I don't use the "test users" for the learning step, how do I get their "latent" representations in order to compute their recommendations ? $\endgroup$ – user3195078 Aug 4 '16 at 15:31
  • $\begingroup$ ^ The second option I've highlighted above should be useful in the case you mentioned (getting latent representations for recommendations). The learning step considers all users, but hides some of their engagement history/implicit preferences. $\endgroup$ – hssay Aug 8 '16 at 4:34
  • $\begingroup$ Yes the second is perfectly fine, I think I'll go for it, thanks again :) $\endgroup$ – user3195078 Aug 10 '16 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.