2
$\begingroup$

Suppose we are given a confusion matrix for a binary classification:

  • tp, fp
  • fn, tn

Now, there are lots of evaluation metrics:

  • POD (probability of detection, aka hit rate, sensitivity, recall, true positive rate) = tp/(tp+fn)
  • Precision = tp/(tp+fp)
  • FAR (false alarm rate) = fp/(tp+fp)
  • CSI (critical success index) = tp/(tp+fn+fp)
  • POFD (probability of false detection) = fp/(tn+fp)
  • ACC (accuracy) = (tp+tn)/(tp+tn+fp+fn)
  • PAG (cant remember)= 1 - fp/(tp+fp)*1
  • F1 = (2*tp/(2*tp+fp+fn))*1
  • ROC-AUC (area under curve) = (tp/(tp+fn)+(1-fp/(fp+tn)))/2
  • SPEC (specificity) = tn/(tn+fp)
  • MCC (Matthews Correlation Coefficient) = (tp * tn - fp * fn) / np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))

Obviously the list could be extended. However, the question is how does one metric relate to the other? Specifically, if one increases, does the other one always increase/decrease?

If the answer is yes, which ones are redundant, i.e. mutually complementary?

On a side note, is choosing just one enough/not enough? How do we know that?

$\endgroup$
  • 1
    $\begingroup$ About your last question: You should choose whichever one(s) express the error rates that most directly affect the practical application of the model. I like ROC because it summarizes the discriminative ability of the model, but it is not always the most interesting metric. $\endgroup$ – Frans Rodenburg Oct 24 at 6:08
  • $\begingroup$ thanks, I think I would go by that. Additionally, I am writing a paper so I need to choose one so maybe comparability with the other similar research is another dimension to consider $\endgroup$ – eugen Oct 24 at 6:10
  • $\begingroup$ Sure, that is a valid reason to prefer a metric, so long as the metric is a good measure of performance. For example, there have been several posts on CV criticizing accuracy in particular. $\endgroup$ – Frans Rodenburg Oct 24 at 6:13
  • $\begingroup$ @FransRodenburg: per my answer, most criticisms of accuracy apply equally to the other metrics (AUROC being a partial exception). Criticisms concentrate on accuracy because it's the most common (mis-)metric. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 24 at 6:15
2
$\begingroup$

I am not aware of any specific redundancies between the metrics you mention.

They do have one commonality, though: most of them are improper scoring rules, can lead to bogus results (especially, but not only, in the case of unbalanced classes), and should not be used. See, for instance:

The one exception is AUC or AUROC, which is at least semi-proper:


Frans Rodenburg asks a very good question in the comments:

Since you mention most should not be used, do you think it is better to use none of those, rather than multiple at once (each of which might highlight different aspects of the model's performance)?

Using multiple measures can run into a kind of schizophrenia. We typically want to use these metrics to pick the best performing algorithm, but what do we do when one algorithm is "best" on metric A, but another one is "best" on metric B? Plus, the incoherence increases when we realize that most algorithms use a completely different metric during training than we might use in evaluating their performance.

I recently commented on this incoherence in the context of time series forecasting (Kolassa, 2019, International Journal of Forecasting).

So my recommendation would be to use none of the above. In particular, we should not calculate binary classifications. Instead, we should aim for full probabilistic predictions and assess these using proper scoring rules. (I go into more details in my answer to the first two threads linked above.)

Of course, we now have the question of which proper scoring rule to choose, since there are multiple ones. Merkle & Steyvers (2013, Decision Analysis) investigates this question.

$\endgroup$
  • $\begingroup$ (+1), but to clarify: Since you mention most should not be used, do you think it is better to use none of those, rather than multiple at once (each of which might highlight different aspects of the model's performance)? $\endgroup$ – Frans Rodenburg Oct 24 at 6:16
  • $\begingroup$ @FransRodenburg: that is a very good question. I have edited my answer. $\endgroup$ – S. Kolassa - Reinstate Monica Oct 24 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.