Accuracy vs. area under the ROC curve

Question

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold setting (the point closest to point (0, 1)), I got the accuracy of the diagnostic system to be 0.8, which is less than the AUC! When I checked the accuracy at another threshold setting which is way far from the optimum threshold I got the accuracy equal to 0.92. Is it possible to get the accuracy of a diagnostic system at the best threshold setting lower than the accuracy at another threshold and also lower than the area under the curve? See the attached picture please.

Answer 1

It's indeed possible. The key is to remember that the accuracy is highly affected by class imbalance. E.g., in your case, you have more negative samples than positive samples, since when the FPR ($=\frac{FP}{FP+TN}$) is close to 0, and TPR (= $\frac{TP}{TP+FN}$) is 0.5, your accuracy ($= \frac{TP+TN}{TP+FN+FP+TN}$) is still very high.

To put it otherwise, since you have many more negative samples, if the classifier predicts 0 all the time, it will still get a high accuracy with FPR and TPR close to 0.

What you call optimum threshold setting (the point closest to point (0, 1)) is just one of many definitions for optimal threshold: it doesn't necessarily optimize the accuracy.

Answer 2

Okay, remember the relation between the $FPR$ (False Positive Rate), $TPR$ (True Positive Rate) and $ACC$ (Accuracy):

$$TPR = \frac{\sum \text{True positive}}{\sum \text{Positive cases}}$$

$$FPR = \frac{\sum \text{False positive}}{\sum \text{Negative cases}}$$

$$ACC = \frac{TPR \cdot \sum \text{Positive cases} + (1-FPR) \cdot \sum \text{Negative cases}}{\sum \text{Positive cases} + \sum \text{Negative cases}}$$

So, $ACC$ can be represented as a weighted average of $TPR$ and $FPR$. If the number of negatives and positives is the same:

$$ACC = \frac{TPR + 1 - FPR}{2}$$

But what if $N_- \gg N_+$? Then: $$ACC(N_- \gg N_+) \approx 1-FPR$$ So, in this case, maximal $ACC$ occurs at minimal $FPR$

---

See this example, negatives outnumber positives 1000:1.

data = c(rnorm(10L), rnorm(10000L)+1)
lab = c(rep(1, 10L), rep(-1, 10000L))
plot(data, lab, col = lab + 3)
tresh = c(-10, data[lab == 1], 10)
do.call(function(x) abline(v = x, col = "gray"), list(tresh))

pred = lapply(tresh, function (x) ifelse(data <= x, 1, -1))
res = data.frame(
  acc = sapply(pred, function(x) sum(x == lab)/length(lab)),
  tpr = sapply(pred, function(x) sum(lab == x & x == 1)/sum(lab == 1)),
  fpr = sapply(pred, function(x) sum(lab != x & x == 1)/sum(lab != 1))
)

res[order(res$acc),]

#> res[order(res$acc),]
#           acc tpr    fpr
#12 0.000999001 1.0 1.0000
#11 0.189110889 1.0 0.8117
#9  0.500099900 0.9 0.5003
#2  0.757742258 0.8 0.2423
#5  0.763136863 0.7 0.2368
#4  0.792007992 0.6 0.2078
#10 0.807292707 0.5 0.1924
#3  0.884215784 0.4 0.1153
#7  0.890709291 0.3 0.1087
#6  0.903096903 0.2 0.0962
#8  0.971428571 0.1 0.0277
#1  0.999000999 0.0 0.0000

See, when fpr is 0 acc is maximum.

And here's the ROC, with accuracy annotated.

plot(sort(res$fpr), sort(res$tpr), type = "S", ylab = "TPR", xlab = "FPR")
text(sort(res$fpr), sort(res$tpr), pos = 4L, lab = round(res$acc[order(res$fpr)], 3L))
abline(a = 0, b = 1)
abline(a = 1, b = -1)

---

The $AUC$ is

1-sum(res$fpr[-12]*0.1)
#[1] 0.74608

---

The bottom line is that you can optimize accuracy in a way resulting in a bogus model (tpr = 0 in my example). That's because accuracy is not a good metric, dichotomization of the result should be left to the decision-maker.

The optimal threshold is said to be the $TPR = 1-FPR$ line because that way both errors have equal weight, even if accuracy is not optimal.

When you have imbalanced classes, optimizing accuracy can be trivial (e.g. predict everyone as the majority class).

---

Another thing, you can't translate most $AUC$ measures to an accuracy estimate like that; see these questions:

• Area under curve of ROC vs. overall accuracy

• Accuracy and area under ROC curve (AUC)

And most important of all: Why is AUC higher for a classifier that is less accurate than for one that is more accurate?