15
$\begingroup$

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold setting (the point closest to point (0, 1)), I got the accuracy of the diagnostic system to be 0.8, which is less than the AUC! When I checked the accuracy at another threshold setting which is way far from the optimum threshold I got the accuracy equal to 0.92. Is it possible to get the accuracy of a diagnostic system at the best threshold setting lower than the accuracy at another threshold and also lower than the area under the curve? See the attached picture please.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Could you please indicate how many samples there were in your analysis? I'm betting it was heavily imbalanced. Also, AUC and accuracy don't translate like that (when you say accuracy is lower than AUC), at all. $\endgroup$ – Firebug Jul 23 '16 at 1:44
  • 1
    $\begingroup$ 269469 are negatives and 37731 are positive; this might be the problem here as per the answers below (the class imbalance). $\endgroup$ – Ali Sultan Jul 23 '16 at 5:46
  • $\begingroup$ keep in mind the problem isn't class imbalance per se, it's the choice of evaluation measure. All in all, $AUC$ is a more reasonable in this scenario, or you could implement balanced accuracy. $\endgroup$ – Firebug Jul 23 '16 at 12:16
  • $\begingroup$ One last thing, if you feel an answer answered your question, you might consider "accepting" the answer (the green check mark). This is not mandatory, but helps the person that answered and also helps the site organization (the question counts as unanswered until you do that), and perhaps people that would make the same question in the future. $\endgroup$ – Firebug Jul 24 '16 at 15:35
8
$\begingroup$

It's indeed possible. The key is to remember that the accuracy is highly affected by class imbalance. E.g., in your case, you have more negative samples than positive samples, since when the FPR ($=\frac{FP}{FP+TN}$) is close to 0, and TPR (= $\frac{TP}{TP+FN}$) is 0.5, your accuracy ($= \frac{TP+TN}{TP+FN+FP+TN}$) is still very high.

To put it otherwise, since you have many more negative samples, if the classifier predicts 0 all the time, it will still get a high accuracy with FPR and TPR close to 0.

What you call optimum threshold setting (the point closest to point (0, 1)) is just one of many definitions for optimal threshold: it doesn't necessarily optimize the accuracy.

enter image description here

$\endgroup$
9
$\begingroup$

Okay, remember the relation between the $FPR$ (False Positive Rate), $TPR$ (True Positive Rate) and $ACC$ (Accuracy):

$$TPR = \frac{\sum \text{True positive}}{\sum \text{Positive cases}}$$

$$FPR = \frac{\sum \text{False positive}}{\sum \text{Negative cases}}$$

$$ACC = \frac{TPR \cdot \sum \text{Positive cases} + (1-FPR) \cdot \sum \text{Negative cases}}{\sum \text{Positive cases} + \sum \text{Negative cases}}$$

So, $ACC$ can be represented as a weighted average of $TPR$ and $FPR$. If the number of negatives and positives is the same:

$$ACC = \frac{TPR + 1 - FPR}{2}$$

But what if $N_- \gg N_+$? Then: $$ACC(N_- \gg N_+) \approx 1-FPR$$ So, in this case, maximal $ACC$ occurs at minimal $FPR$


See this example, negatives outnumber positives 1000:1.

data = c(rnorm(10L), rnorm(10000L)+1)
lab = c(rep(1, 10L), rep(-1, 10000L))
plot(data, lab, col = lab + 3)
tresh = c(-10, data[lab == 1], 10)
do.call(function(x) abline(v = x, col = "gray"), list(tresh))

pred = lapply(tresh, function (x) ifelse(data <= x, 1, -1))
res = data.frame(
  acc = sapply(pred, function(x) sum(x == lab)/length(lab)),
  tpr = sapply(pred, function(x) sum(lab == x & x == 1)/sum(lab == 1)),
  fpr = sapply(pred, function(x) sum(lab != x & x == 1)/sum(lab != 1))
)

res[order(res$acc),]

#> res[order(res$acc),]
#           acc tpr    fpr
#12 0.000999001 1.0 1.0000
#11 0.189110889 1.0 0.8117
#9  0.500099900 0.9 0.5003
#2  0.757742258 0.8 0.2423
#5  0.763136863 0.7 0.2368
#4  0.792007992 0.6 0.2078
#10 0.807292707 0.5 0.1924
#3  0.884215784 0.4 0.1153
#7  0.890709291 0.3 0.1087
#6  0.903096903 0.2 0.0962
#8  0.971428571 0.1 0.0277
#1  0.999000999 0.0 0.0000

See, when fpr is 0 acc is maximum.

And here's the ROC, with accuracy annotated.

plot(sort(res$fpr), sort(res$tpr), type = "S", ylab = "TPR", xlab = "FPR")
text(sort(res$fpr), sort(res$tpr), pos = 4L, lab = round(res$acc[order(res$fpr)], 3L))
abline(a = 0, b = 1)
abline(a = 1, b = -1)

enter image description here


The $AUC$ is

1-sum(res$fpr[-12]*0.1)
#[1] 0.74608

The bottom line is that you can optimize accuracy in a way resulting in a bogus model (tpr = 0 in my example). That's because accuracy is not a good metric, dichotomization of the result should be left to the decision-maker.

The optimal threshold is said to be the $TPR = 1-FPR$ line because that way both errors have equal weight, even if accuracy is not optimal.

When you have imbalanced classes, optimizing accuracy can be trivial (e.g. predict everyone as the majority class).


Another thing, you can't translate most $AUC$ measures to an accuracy estimate like that; see these questions:

And most important of all: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.