Which evaluation metrics are mutually redundant?

Question

Suppose we are given a confusion matrix for a binary classification:

• tp, fp
• fn, tn

Now, there are lots of evaluation metrics:

• POD (probability of detection, aka hit rate, sensitivity, recall, true positive rate) = tp/(tp+fn)
• Precision = tp/(tp+fp)
• FAR (false alarm rate) = fp/(tp+fp)
• CSI (critical success index) = tp/(tp+fn+fp)
• POFD (probability of false detection) = fp/(tn+fp)
• ACC (accuracy) = (tp+tn)/(tp+tn+fp+fn)
• PAG (cant remember)= 1 - fp/(tp+fp)*1
• F1 = (2*tp/(2*tp+fp+fn))*1
• ROC-AUC (area under curve) = (tp/(tp+fn)+(1-fp/(fp+tn)))/2
• SPEC (specificity) = tn/(tn+fp)
• MCC (Matthews Correlation Coefficient) = (tp * tn - fp * fn) / np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))

Obviously the list could be extended. However, the question is how does one metric relate to the other? Specifically, if one increases, does the other one always increase/decrease?

If the answer is yes, which ones are redundant, i.e. mutually complementary?

On a side note, is choosing just one enough/not enough? How do we know that?

Answer 1

I am not aware of any specific redundancies between the metrics you mention.

They do have one commonality, though: most of them are improper scoring rules, can lead to bogus results (especially, but not only, in the case of unbalanced classes), and should not be used. See, for instance:

• Why is accuracy not the best measure for assessing classification models? - Everything noted here also applies to the other metrics
• Classification probability threshold

The one exception is AUC or AUROC, which is at least semi-proper:

• What does it mean that AUC is a semi-proper scoring rule?
• Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

---

Frans Rodenburg asks a very good question in the comments:

Since you mention most should not be used, do you think it is better to use none of those, rather than multiple at once (each of which might highlight different aspects of the model's performance)?

Using multiple measures can run into a kind of schizophrenia. We typically want to use these metrics to pick the best performing algorithm, but what do we do when one algorithm is "best" on metric A, but another one is "best" on metric B? Plus, the incoherence increases when we realize that most algorithms use a completely different metric during training than we might use in evaluating their performance.

I recently commented on this incoherence in the context of time series forecasting (Kolassa, 2019, International Journal of Forecasting).

So my recommendation would be to use none of the above. In particular, we should not calculate binary classifications. Instead, we should aim for full probabilistic predictions and assess these using proper scoring rules. (I go into more details in my answer to the first two threads linked above.)

Of course, we now have the question of which proper scoring rule to choose, since there are multiple ones. Merkle & Steyvers (2013, Decision Analysis) investigates this question.