I'm trying to build a recommendation system, and have a bunch of (item,item_features,liked) triplets, where liked is binary. Most items are not liked. So I'm running a logistic regression with glmnet of the form liked ~ item_features This yields an AUC of around 0.75 (it doesn't vary much with the regularization parameter). However, the error rate (also doesn't vary much) is only a tiny,tiny bit better than what you would get if you just always predicted "don't like." What is the best way (or any way, really!) to think about the value or lack thereof this recommender?

  • $\begingroup$ What is a proportion of likes and not likes in your response? $\endgroup$
    – user88
    Jun 30, 2012 at 0:21
  • $\begingroup$ @alex I wonder what the purpose of this system is. Such a system without any context (may it be a specific user or another item) will only recommend the items the most people liked, won't it ? What happens if one calculates the $\frac{like}{dislike}-ratio$ per item and sort the items in exactly that manner ? Am I right with the assumption that the same items appears multiple times, maybe with likes AND dislikes ? The only purpose I can imagine is to get some knowledge why items are liked ... $\endgroup$
    – steffen
    Aug 29, 2012 at 11:55
  • $\begingroup$ @alex another question: Are the 0-ratings implicit or explicit, i.e. explicit means "user has explicitly disliked it (by e.g. clicking on a dislike button) meanwhile implict that user has not just not expressed a positive opinion (by e.g. clicking on the like-button). $\endgroup$
    – steffen
    Aug 29, 2012 at 11:57
  • $\begingroup$ When looking at the error rate, what is your cutoff? $\endgroup$
    – Erik
    Aug 29, 2012 at 13:01
  • 1
    $\begingroup$ Error rate is an example of an improper scoring rule - a measure that is optimized by a bogus model. So I would never use proportion classified correct. $\endgroup$ Aug 29, 2012 at 21:41

1 Answer 1


0.75 is a pretty modest discrimination, I would say that's a decent model in the case of recommender systems where the volume of material someone is exposured to is gigantic relative to what the encounter and eventually like. You'd expect only an incremental positive predictive value in that case, since the outcome is rare, so discounting the model for having low PPV would be too harsh.

  • $\begingroup$ In general I do agree, but what is PPV ? $\endgroup$
    – steffen
    Aug 29, 2012 at 11:36
  • $\begingroup$ Positive predictive value. Unlike sensitivity / specificity which measure Pr(Test + | Disease), this measures Pr(Disease | Test +) which is more useful for forecasting / planning. You can have very strong recall in tests that has very modest/weak discrimination and predictive value. For comparing predictive models (you chose always "don't like" as an example), you can go far comparing AUC. $\endgroup$
    – AdamO
    Sep 6, 2012 at 17:28
  • $\begingroup$ thank you. I know this metric by the name precision. $\endgroup$
    – steffen
    Sep 7, 2012 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.