What metric should I use for assessing Implicit matrix factorization recommender with ALS?

Question

I'm currently implementing a recommender system using implicit ratings (time spent on an article) but I'm wondering what would be the proper metric to assess the system. MSE doesn't seems to fit to this usecase.

My algorithm is based on the paper : Collaborative Filtering for Implicit Feedback Datasets

Answer 1

Evaluating implicit feedback based recommendations is tricky. Here are couple of approaches that I'd recommend.

Approach 1

You can use modified precision and recall metrics. Overall the procedure would be as follows,

1. Divide your data into train and test set by users.
2. For a user in test set, given their history, get the top N recommendations using implicit feedback based model.
3. Precision can be calculated using # of recommendations given by model which actually matched by what user had acted upon (for example read in case of articles).
4. Recall can be calculated using # of user actions (articles read by user) that were captured by top N recommendations.
5. You can calculate these for all users in test set and average them.

Approach 2

The approach is similar to approach 1, but rather than splitting train and test data by users, you use something called as "leave one out" strategy. We use a simple accuracy metric in this case. The steps to calculate the accuracy will be as follows,

1. For each user (or a subset of users), hide one of the articles read/browsed and move it to test set (leave one out).
2. Using user history, get top N recommendations for each user.
3. Calculate the number of times the left out article was captured by the top N recommendations.

Note that in both cases, the N in top N recommendations will become your hyper-parameter which can be tuned further.