I'm trying to build a recommendation system, and have a bunch of (item,item_features,liked) triplets, where liked is binary. Most items are not liked. So I'm running a logistic regression with glmnet of the form
liked ~ item_features
This yields an AUC of around 0.75 (it doesn't vary much with the regularization parameter). However, the error rate (also doesn't vary much) is only a tiny,tiny bit better than what you would get if you just always predicted "don't like." What is the best way (or any way, really!) to think about the value or lack thereof this recommender?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ What is a proportion of likes and not likes in your response? $\endgroup$– user88Jun 30, 2012 at 0:21
-
$\begingroup$ @alex I wonder what the purpose of this system is. Such a system without any context (may it be a specific user or another item) will only recommend the items the most people liked, won't it ? What happens if one calculates the $\frac{like}{dislike}-ratio$ per item and sort the items in exactly that manner ? Am I right with the assumption that the same items appears multiple times, maybe with likes AND dislikes ? The only purpose I can imagine is to get some knowledge why items are liked ... $\endgroup$– steffenAug 29, 2012 at 11:55
-
$\begingroup$ @alex another question: Are the 0-ratings implicit or explicit, i.e. explicit means "user has explicitly disliked it (by e.g. clicking on a dislike button) meanwhile implict that user has not just not expressed a positive opinion (by e.g. clicking on the like-button). $\endgroup$– steffenAug 29, 2012 at 11:57
-
$\begingroup$ When looking at the error rate, what is your cutoff? $\endgroup$– ErikAug 29, 2012 at 13:01
-
1$\begingroup$ Error rate is an example of an improper scoring rule - a measure that is optimized by a bogus model. So I would never use proportion classified correct. $\endgroup$– Frank HarrellAug 29, 2012 at 21:41
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
0.75 is a pretty modest discrimination, I would say that's a decent model in the case of recommender systems where the volume of material someone is exposured to is gigantic relative to what the encounter and eventually like. You'd expect only an incremental positive predictive value in that case, since the outcome is rare, so discounting the model for having low PPV would be too harsh.
-
-
$\begingroup$ Positive predictive value. Unlike sensitivity / specificity which measure Pr(Test + | Disease), this measures Pr(Disease | Test +) which is more useful for forecasting / planning. You can have very strong recall in tests that has very modest/weak discrimination and predictive value. For comparing predictive models (you chose always "don't like" as an example), you can go far comparing AUC. $\endgroup$– AdamOSep 6, 2012 at 17:28
-