# Which evaluation metrics are mutually redundant?

Suppose we are given a confusion matrix for a binary classification:

• tp, fp
• fn, tn

Now, there are lots of evaluation metrics:

• POD (probability of detection, aka hit rate, sensitivity, recall, true positive rate) = tp/(tp+fn)
• Precision = tp/(tp+fp)
• FAR (false alarm rate) = fp/(tp+fp)
• CSI (critical success index) = tp/(tp+fn+fp)
• POFD (probability of false detection) = fp/(tn+fp)
• ACC (accuracy) = (tp+tn)/(tp+tn+fp+fn)
• PAG (cant remember)= 1 - fp/(tp+fp)*1
• F1 = (2*tp/(2*tp+fp+fn))*1
• ROC-AUC (area under curve) = (tp/(tp+fn)+(1-fp/(fp+tn)))/2
• SPEC (specificity) = tn/(tn+fp)
• MCC (Matthews Correlation Coefficient) = (tp * tn - fp * fn) / np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))

Obviously the list could be extended. However, the question is how does one metric relate to the other? Specifically, if one increases, does the other one always increase/decrease?

If the answer is yes, which ones are redundant, i.e. mutually complementary?

On a side note, is choosing just one enough/not enough? How do we know that?

• About your last question: You should choose whichever one(s) express the error rates that most directly affect the practical application of the model. I like ROC because it summarizes the discriminative ability of the model, but it is not always the most interesting metric. Oct 24 '19 at 6:08
• thanks, I think I would go by that. Additionally, I am writing a paper so I need to choose one so maybe comparability with the other similar research is another dimension to consider Oct 24 '19 at 6:10
• Sure, that is a valid reason to prefer a metric, so long as the metric is a good measure of performance. For example, there have been several posts on CV criticizing accuracy in particular. Oct 24 '19 at 6:13
• @FransRodenburg: per my answer, most criticisms of accuracy apply equally to the other metrics (AUROC being a partial exception). Criticisms concentrate on accuracy because it's the most common (mis-)metric. Oct 24 '19 at 6:15

I am not aware of any specific redundancies between the metrics you mention.

They do have one commonality, though: most of them are improper scoring rules, can lead to bogus results (especially, but not only, in the case of unbalanced classes), and should not be used. See, for instance:

The one exception is AUC or AUROC, which is at least semi-proper: