Implementing Recommender System Ranking Algorithm in Python

Question

I am trying to do collaborative filtering for implicit feedback datasets by following the seminal paper:

http://yifanhu.net/PUB/cf.pdf

The section on ranking says:

I have a matrix of 50K X 9K with each cell having no. of times a user has seen a video. I also have a binary matrix of the same (watched or not; 1 or 0)

If I divide the dataset into 80:20 in training and test, and run the Recommender algorithm on 80% training, how do I evaluate the above mentioned ranking algo on the test in Python. I guess I am having a bit of trouble in understanding the algo's implementation.

Any help?

Answer 1

The parameters of the matrix factorization algorithm you link to above are the user and item latent vectors. The dot product of any user vector and the item latent vector matrix yields recommendation scores for all items for that particular user; when you treat those as defining a preference ranking over items, you can use standard ranking evaluation metrics to measure the quality of your recommender.

The procedure would go roughly as follows:

1. Fit the model.
2. Iterate over all users in your test set; for every user compute their recommendations by multiplying their user vector with the item vectors.
3. Using this predicted ranking, and the known positive items for that user from the hold-out test set, compute your choice of ranking metric. Precision@K and mean reciprocal rank are two metrics that are particularly easy to interpret and common in the literature. Reciprocal rank is the inverse of the metric defined in the section you quote when you set the weights $r_{ui}$ to 1.
4. To evaluate the model as a whole, average your metric over all the users in your test set.

During evaluation, you may want to exclude the known positives from the training set. This would correspond to a setting where known positive items are excluded from further recommendations (because the user has already bought or seen them).