7
$\begingroup$

Just read a unpublished paper for review purpose. The reported results like

Leave-one-out cross validation sensitivity is 95%. Leave-one-out cross validation specificity is 100%. Leave-one-out cross validation AUC is 1.

Is this possible?

If AUC is 1, does that mean both sensitivity and specificity should be 100%? Just can't get my head around how they got this kind of results.

If it is possible then how to interpret the results?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Beware that the analyst is hiding steps (e.g., feature selection) from the LOO-CV. That would make all these indexes biased. Also I'm not convinced the LOO-CV pays the full penalty for feature selection because any two samples have an overlap of $n-2$ observations. $\endgroup$
    – Frank Harrell
    May 12, 2014 at 17:31
  • 1
    $\begingroup$ @FrankHarrell: I fully agree with data-driven model selection without outer independent validation being a probem. But how do you know that's the main problem here? (I have a longer check list of common problems that lead to heavy optimistic bias in validation such as leave-one-measurement-out instead of leave-one-patient-out, all kinds of data-driven steps not re-calculated for each surrogate model; and most of them happen almost as easily with supposedly "independent" test sets). $\endgroup$
    – cbeleites unhappy with SX
    May 13, 2014 at 5:52
  • 1
    $\begingroup$ I am still not clear on whether the OP repeated all modeling steps that used $Y$ when doing LOO. LOO requires $n$ model re-developments. $\endgroup$
    – Frank Harrell
    May 13, 2014 at 12:08

2 Answers 2

Reset to default
10
$\begingroup$

An alternative interpretation of AUC is that it gives the probability that a randomly selected positive is ranked above a randomly selected negative (see, e.g., Wikipedia). Thus, AUC is 1 if and only if all positives are ranked above all negatives. However, it is still possible to obtain sensitivity or specificity under 1 by selecting a suboptimal cutoff.

For example, let us consider a test case with 4 subjects and an algorithm predicting probabilities (of being positive) as follows: \begin{equation} \begin{array}{c|c} \textrm{Truth} & \textrm{Probability} \\ \hline \textrm{P} & 0.9 \\ \textrm{P} & 0.4 \\ \textrm{N} & 0.2 \\ \textrm{N} & 0.1 \end{array} \end{equation} All positives are ranked above all negatives, thus the AUC is 1. However, if the cutoff probability is set as $0.5$, one of the positives is classified as negative, and thus the sensitivity is only $50\%$. Indeed, depending on the cutoff, the sensitivities and false positive rates ($1-$ specifities) will be as follows. \begin{equation} \begin{array}{c|c|c} \textrm{Cutoff in range} & \textrm{Sensitivity} & 1-\textrm{ Specificity} \\ \hline (-\infty,0.1] & 1 & 1 \\ (0.1,0.2] & 1 & 0.5 \\ (0.2,0.4] & 1 & 0 \\ (0.4,0.9] & 0.5 & 0 \\ (0.9,\infty) & 0 & 0 \end{array} \end{equation}

This is illustrated in the following figure. The possible (Sensitivity, $1-$ Specificity) combinations are drawn as red circles and the (interpolated) ROC curve as green line. The entire unit square is under the curve, and thus the area under the curve is 1.

ROC curve of the example

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ why will you want to select a sub-optimal when you indeed know the whole ROC? $\endgroup$
    – Chamberlain Mbah
    May 13, 2014 at 6:30
-1
$\begingroup$

AUC=1 means for all specificities in [0,1] the sensitivity is 1. It is IMPOSSIBLE to get a sensitivity of 95%. So that paper is reporting a fundamental error. Hope this helps.

Improve this answer
$\endgroup$
12
  • 1
    $\begingroup$ This is wrong. See @Juho's example. $\endgroup$
    – cbeleites unhappy with SX
    May 13, 2014 at 5:45
  • $\begingroup$ Like I already asked @Juho, why will you want to take a sub-optimal point when indeed there is an optimal one. My answer says for every specificity in [0,1] there is a sensitivity of 1 corresponding to it. Why then go for other sensitivities at specificity of 1 when there exist a sensitivity of 1. Think of the ultimate reason of looking at the ROC curve. $\endgroup$
    – Chamberlain Mbah
    May 13, 2014 at 6:42
  • 2
    $\begingroup$ One very good reason is because the working point was chosen previously: if the results in question are from the final model's validation, no further data-driven tuning (neither of the cutoff/ROC working point nor of any other hyperparameter) is allowed. $\endgroup$
    – cbeleites unhappy with SX
    May 13, 2014 at 10:52
  • 1
    $\begingroup$ @ChamberlainFoncha: well it is only called receiver operating curve, not receiver operating function, and I'd rather say the relationship is sensitivity = f (model, cutoff) and specificity = f (model, cutoff), so (1 - specificity; sensitivity) = f (model, cutoff). Moreover, multiple points at 1 - specificity = 0 frequently arise in a "natural" way, see Juho's description of the calculations. Why do you think the ROC should itself be a function (as opposed to a curve)? And if sensitivity = f (specificity), why not specificity = f (sensitivity)? $\endgroup$
    – cbeleites unhappy with SX
    May 14, 2014 at 15:55
  • 1
    $\begingroup$ Indeed you are right.... $\endgroup$
    – Chamberlain Mbah
    May 14, 2014 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.