Suppose we are given a confusion matrix for a binary classification:
- tp, fp
- fn, tn
Now, there are lots of evaluation metrics:
- POD (probability of detection, aka hit rate, sensitivity, recall, true positive rate) = tp/(tp+fn)
- Precision = tp/(tp+fp)
- FAR (false alarm rate) = fp/(tp+fp)
- CSI (critical success index) = tp/(tp+fn+fp)
- POFD (probability of false detection) = fp/(tn+fp)
- ACC (accuracy) = (tp+tn)/(tp+tn+fp+fn)
- PAG (cant remember)= 1 - fp/(tp+fp)*1
- F1 = (2*tp/(2*tp+fp+fn))*1
- ROC-AUC (area under curve) = (tp/(tp+fn)+(1-fp/(fp+tn)))/2
- SPEC (specificity) = tn/(tn+fp)
- MCC (Matthews Correlation Coefficient) = (tp * tn - fp * fn) / np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
Obviously the list could be extended. However, the question is how does one metric relate to the other? Specifically, if one increases, does the other one always increase/decrease?
If the answer is yes, which ones are redundant, i.e. mutually complementary?
On a side note, is choosing just one enough/not enough? How do we know that?