Accuracy, Sensitivity, Specificity, & ROC AUC [duplicate]

Question

In the context of predictive modeling, when comparing clasification models, What statistic should be considered more important over the others: Accuracy, sensitivity, specificity, or area under ROC curve? Would you suggest other statistics?

Thanks & regards.

Answer 1

In case if dataset is imbalanced.

I would suggest to use a Precision & Recall metric or Precision | Recall curve instead of ROC and Accuracy.

[https://www.kaggle.com/general/7517]

[https://classeval.wordpress.com/simulation-analysis/roc-and-precision-recall-with-imbalanced-datasets/]

The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Answer 2

Accuracy, sensitivity and specificity are improper scoring rules. They all have major problems in unbalanced datasets, and almost as big problems in balanced datasets. See Why is accuracy not the best measure for assessing classification models?

AUROC is slightly better, it is a semi-proper scoring rule: What does it mean that AUC is a semi-proper scoring rule?

The best approach is to use probabilistic predictions and proper scoring rules. See my answer to the thread cited above.

Answer 3

For an apples to apples comparison, the area under the ROC (AUC) would be the best metric. This is because the AUC does not depend on the thresholding value. It is also not sensitive to imbalances in the dataset. (Ideally, we should use the same validation/test data to perform comparisons, so one could argue that the dataset imbalance is not such a big deal.)

Depending on the application, the sensitivity or specificity might be more important. For example, you might have high penalties for false negatives, which implies you want high sensitivity but can tolerate some loss in specificity. In such cases, it will make sense to check what is the best sensitivity (or specificity) you can achieve, this can be obtained from the ROC (the complete curve). The AUC might be misleading, in such cases.

For alternate metrics, you could also consider the Precision-Recall (Recall = Sensitivity) curves (see https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432). Note however that Precision is affected by data imbalance.

---

There is some good discussion here:

https://lukeoakdenrayner.wordpress.com/2017/12/06/do-machines-actually-beat-doctors-roc-curves-and-performance-metrics/

https://lukeoakdenrayner.wordpress.com/2018/01/07/the-philosophical-argument-for-using-roc-curves/

https://www.site.uottawa.ca/~stan/csi7162/presentations/William-presentation.pdf