Just read a unpublished paper for review purpose. The reported results like

Leave-one-out cross validation sensitivity is 95%. Leave-one-out cross validation specificity is 100%. Leave-one-out cross validation AUC is 1.

Is this possible?

If AUC is 1, does that mean both sensitivity and specificity should be 100%? Just can't get my head around how they got this kind of results.

If it is possible then how to interpret the results?

  • 3
    $\begingroup$ Beware that the analyst is hiding steps (e.g., feature selection) from the LOO-CV. That would make all these indexes biased. Also I'm not convinced the LOO-CV pays the full penalty for feature selection because any two samples have an overlap of $n-2$ observations. $\endgroup$ – Frank Harrell May 12 '14 at 17:31
  • 1
    $\begingroup$ @FrankHarrell: I fully agree with data-driven model selection without outer independent validation being a probem. But how do you know that's the main problem here? (I have a longer check list of common problems that lead to heavy optimistic bias in validation such as leave-one-measurement-out instead of leave-one-patient-out, all kinds of data-driven steps not re-calculated for each surrogate model; and most of them happen almost as easily with supposedly "independent" test sets). $\endgroup$ – cbeleites supports Monica May 13 '14 at 5:52
  • 1
    $\begingroup$ I am still not clear on whether the OP repeated all modeling steps that used $Y$ when doing LOO. LOO requires $n$ model re-developments. $\endgroup$ – Frank Harrell May 13 '14 at 12:08

An alternative interpretation of AUC is that it gives the probability that a randomly selected positive is ranked above a randomly selected negative (see, e.g., Wikipedia). Thus, AUC is 1 if and only if all positives are ranked above all negatives. However, it is still possible to obtain sensitivity or specificity under 1 by selecting a suboptimal cutoff.

For example, let us consider a test case with 4 subjects and an algorithm predicting probabilities (of being positive) as follows: \begin{equation} \begin{array}{c|c} \textrm{Truth} & \textrm{Probability} \\ \hline \textrm{P} & 0.9 \\ \textrm{P} & 0.4 \\ \textrm{N} & 0.2 \\ \textrm{N} & 0.1 \end{array} \end{equation} All positives are ranked above all negatives, thus the AUC is 1. However, if the cutoff probability is set as $0.5$, one of the positives is classified as negative, and thus the sensitivity is only $50\%$. Indeed, depending on the cutoff, the sensitivities and false positive rates ($1-$ specifities) will be as follows. \begin{equation} \begin{array}{c|c|c} \textrm{Cutoff in range} & \textrm{Sensitivity} & 1-\textrm{ Specificity} \\ \hline (-\infty,0.1] & 1 & 1 \\ (0.1,0.2] & 1 & 0.5 \\ (0.2,0.4] & 1 & 0 \\ (0.4,0.9] & 0.5 & 0 \\ (0.9,\infty) & 0 & 0 \end{array} \end{equation}

This is illustrated in the following figure. The possible (Sensitivity, $1-$ Specificity) combinations are drawn as red circles and the (interpolated) ROC curve as green line. The entire unit square is under the curve, and thus the area under the curve is 1.

ROC curve of the example

  • 1
    $\begingroup$ why will you want to select a sub-optimal when you indeed know the whole ROC? $\endgroup$ – Chamberlain Foncha May 13 '14 at 6:30

AUC=1 means for all specificities in [0,1] the sensitivity is 1. It is IMPOSSIBLE to get a sensitivity of 95%. So that paper is reporting a fundamental error. Hope this helps.

  • 1
    $\begingroup$ This is wrong. See @Juho's example. $\endgroup$ – cbeleites supports Monica May 13 '14 at 5:45
  • $\begingroup$ Like I already asked @Juho, why will you want to take a sub-optimal point when indeed there is an optimal one. My answer says for every specificity in [0,1] there is a sensitivity of 1 corresponding to it. Why then go for other sensitivities at specificity of 1 when there exist a sensitivity of 1. Think of the ultimate reason of looking at the ROC curve. $\endgroup$ – Chamberlain Foncha May 13 '14 at 6:42
  • 2
    $\begingroup$ One very good reason is because the working point was chosen previously: if the results in question are from the final model's validation, no further data-driven tuning (neither of the cutoff/ROC working point nor of any other hyperparameter) is allowed. $\endgroup$ – cbeleites supports Monica May 13 '14 at 10:52
  • $\begingroup$ Also note that AUC = 1 means that for specificity in (0, 1] sensitivity = 1 and for sensitivity in (0, 1], specificity = 1 $\endgroup$ – cbeleites supports Monica May 13 '14 at 10:53
  • 1
    $\begingroup$ Indeed you are right.... $\endgroup$ – Chamberlain Foncha May 14 '14 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.