2
$\begingroup$

I'm trying to build a recommendation system, and have a bunch of (item,item_features,liked) triplets, where liked is binary. Most items are not liked. So I'm running a logistic regression with glmnet of the form liked ~ item_features This yields an AUC of around 0.75 (it doesn't vary much with the regularization parameter). However, the error rate (also doesn't vary much) is only a tiny,tiny bit better than what you would get if you just always predicted "don't like." What is the best way (or any way, really!) to think about the value or lack thereof this recommender?

$\endgroup$
  • $\begingroup$ What is a proportion of likes and not likes in your response? $\endgroup$ – mbq Jun 30 '12 at 0:21
  • $\begingroup$ @alex I wonder what the purpose of this system is. Such a system without any context (may it be a specific user or another item) will only recommend the items the most people liked, won't it ? What happens if one calculates the $\frac{like}{dislike}-ratio$ per item and sort the items in exactly that manner ? Am I right with the assumption that the same items appears multiple times, maybe with likes AND dislikes ? The only purpose I can imagine is to get some knowledge why items are liked ... $\endgroup$ – steffen Aug 29 '12 at 11:55
  • $\begingroup$ @alex another question: Are the 0-ratings implicit or explicit, i.e. explicit means "user has explicitly disliked it (by e.g. clicking on a dislike button) meanwhile implict that user has not just not expressed a positive opinion (by e.g. clicking on the like-button). $\endgroup$ – steffen Aug 29 '12 at 11:57
  • $\begingroup$ When looking at the error rate, what is your cutoff? $\endgroup$ – Erik Aug 29 '12 at 13:01
  • 1
    $\begingroup$ Error rate is an example of an improper scoring rule - a measure that is optimized by a bogus model. So I would never use proportion classified correct. $\endgroup$ – Frank Harrell Aug 29 '12 at 21:41
1
$\begingroup$

0.75 is a pretty modest discrimination, I would say that's a decent model in the case of recommender systems where the volume of material someone is exposured to is gigantic relative to what the encounter and eventually like. You'd expect only an incremental positive predictive value in that case, since the outcome is rare, so discounting the model for having low PPV would be too harsh.

$\endgroup$
  • $\begingroup$ In general I do agree, but what is PPV ? $\endgroup$ – steffen Aug 29 '12 at 11:36
  • $\begingroup$ Positive predictive value. Unlike sensitivity / specificity which measure Pr(Test + | Disease), this measures Pr(Disease | Test +) which is more useful for forecasting / planning. You can have very strong recall in tests that has very modest/weak discrimination and predictive value. For comparing predictive models (you chose always "don't like" as an example), you can go far comparing AUC. $\endgroup$ – AdamO Sep 6 '12 at 17:28
  • $\begingroup$ thank you. I know this metric by the name precision. $\endgroup$ – steffen Sep 7 '12 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.