0
$\begingroup$

I have 2 churn prediction model. Both provided very similar AUC values for Roc but with different shape. How should I assess which to choose based on that fact?

$\endgroup$
  • 2
    $\begingroup$ Depending on what is more important to you, Recall or Precision. $\endgroup$ – user2974951 Sep 3 at 12:02
  • $\begingroup$ You mean sensitivity and specificity. This isn't a PR curve. $\endgroup$ – Christopher John Oct 23 at 12:04
  • $\begingroup$ @ChristopherJohn I mean 2 Roc curves. $\endgroup$ – Luis Oct 27 at 18:24
0
$\begingroup$

Many researchers have tried to make decisions on the basis of shape differences in ROC curves, but in my view this has been futile. That is because the points on an ROC represent backwards-information-flow probabilities, i.e., probability that a predictor is greater than a threshold given the Y=0 or given Y=1. Each point on the curve is an improper scoring rule. And areas under ROC are not sensitive enough measures for comparing two models. For that purpose you should use a proper accuracy scoring rule such as the Brier score, or use one of the measures described here.

$\endgroup$
0
$\begingroup$

Typically, the “business use” of a classifier depends on sensitivity and specificity, not on AUC. What you have is to model with same average performance (what the AUC measures) but one is optimal at high sensitivity/low specificity, and the other is optimal in the opposite use case.

So the right question to ask is: what is the quantity I want to maximise/minimise? Clearly AUC does not tell you anything about it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.