Recommender system implicit ratings evaluation

Question

I have a music recommender system with implicit ratings on Apache Spark with MLlib, CF and ALS. I have several ways of how I get preference matrix from raw events data.

Now I just count how many times each user played each song. But I want rating to decrease with time after last song playing. I assumed two strategies: rating decline linearly and hyperbolic with days after last event.

How can I evaluate these two strategies offline on raw data without A/B testing or similar techniques that involve user interaction? How can I make an approximation for parameters?

Answer 1

Your question seems two-pronged, so I am going to provide two answers.

1. Evaluation: recommender systems are normally evaluated using ranking metrics on a hold-out set of interactions. In an application where the passage of time seems important, you can divide your interaction data into a training set containing preference information up to a certain time, and a test set containing interactions after that time. You then train your model on the training data, and use it to compute preference rankings for each user: a list of items ordered from most to least recommended. If the interactions you have in your test set rank high in those rankings, you likely have a good model. Common metrics used to evaluate this include precision@k and mean reciprocal rank (MRR). This offline evaluation cannot replace online testing, but it will allow you to select a shortlist of models you would like to promote to A/B testing.
2. Model formulation: you can encode your decay scheme by weighting the training interactions as you feed them into your model. In the ALS-WR algorithm (Hu et al., Collaborative Filtering for Implicit Feedback Datasets), you can use confidence weights that decay with the age of the interaction. In stochastic gradient descent algorithms, you can use sample weights to affect how much importance is given to each interaction during fitting.