2
$\begingroup$

I have a music recommender system with implicit ratings on Apache Spark with MLlib, CF and ALS. I have several ways of how I get preference matrix from raw events data.

Now I just count how many times each user played each song. But I want rating to decrease with time after last song playing. I assumed two strategies: rating decline linearly and hyperbolic with days after last event.

How can I evaluate these two strategies offline on raw data without A/B testing or similar techniques that involve user interaction? How can I make an approximation for parameters?

$\endgroup$
2
$\begingroup$

Your question seems two-pronged, so I am going to provide two answers.

  1. Evaluation: recommender systems are normally evaluated using ranking metrics on a hold-out set of interactions. In an application where the passage of time seems important, you can divide your interaction data into a training set containing preference information up to a certain time, and a test set containing interactions after that time. You then train your model on the training data, and use it to compute preference rankings for each user: a list of items ordered from most to least recommended. If the interactions you have in your test set rank high in those rankings, you likely have a good model. Common metrics used to evaluate this include precision@k and mean reciprocal rank (MRR). This offline evaluation cannot replace online testing, but it will allow you to select a shortlist of models you would like to promote to A/B testing.
  2. Model formulation: you can encode your decay scheme by weighting the training interactions as you feed them into your model. In the ALS-WR algorithm (Hu et al., Collaborative Filtering for Implicit Feedback Datasets), you can use confidence weights that decay with the age of the interaction. In stochastic gradient descent algorithms, you can use sample weights to affect how much importance is given to each interaction during fitting.
$\endgroup$
  • $\begingroup$ Please note if you want to promote your own product/blog you must disclose your affiliation, otherwise your answer may be flagged as spam. Please read How to not be a spammer $\endgroup$ – DavidPostill Aug 9 '17 at 15:57
  • 2
    $\begingroup$ You don't need to remove the link. Just say it's your work :) $\endgroup$ – DavidPostill Aug 9 '17 at 16:04
  • $\begingroup$ Well, it wasn't the most helpful link of all the ones that I could have included. I'll claw my way to higher reputation and link papers and measure definitions first. $\endgroup$ – Maciej Kula Aug 10 '17 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.