0
$\begingroup$

If I have a good measure of the AUC, what does it tell me in relation to the model and its variables?

$\endgroup$
4
$\begingroup$

No.

  1. The AUC is a measure of discrimination. Say I only fit smoking (Y/N) in a risk model for lung cancer. If I predict 100% risk in smokers who have a 0.4% event rate and 0% risk in a non-smokers who have a 0.01% event rate, the model is optimal in terms of its AUC.

  2. A good measure of AUC should take account of overfitting, using bootstrap or cross-validation, e.g. external validation. Model fit statistics are be definition a measure of internal validation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, that's a really good explanation. $\endgroup$ – Omar Charming Khodr Nov 26 '19 at 1:54
1
$\begingroup$

There is no single number that captures the quality of a classifier: the whole path of the ROC contains more information than any single statistic. A perfect classifier has an area of 1.0 under the ROC, and a classifier with a large AUROC is good. This is especially true if the classes are balanced.

If you are classifying with unbalanced classes, e.g. screening for a few possible drugs among millions of candidates, then a big change in performance can result in only a small change in AUROC. In such cases, the PR curve (precision versus recall) is more informative to the eye than the ROC curve, although mathematically it contains the same information. In such cases, Area Under PR would be a better summary statistic than AUROC.

If you deal with unbalanced classifiers, it is worth getting used to reading PR curves. It is sensible to report ROC as well, since more people are used to reading them.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.