I am working on a a binary classification problem with the following qualities:
- Highly imbalanced
- False Negative errors much more costly than False Positive
- False Positives are only False until investigated; may become True Positives
- True Negatives are by far the largest class
- Spatial in nature (raster based)
I am searching for a threshold dependent binary classification metric that utilizes the quandrant of a confusion matrix (TP,FP,TN,FN) to assess the magnitude of True Positives realtive to the area of False Positives. Preferably, this metric would use only Sensitivity and Specificity as these are the more commonly reported quantities in previous studies. Below are examples of metrics that fit some requirements, but not all.
The use of classification metrics and confusion matrices is not overly common in my problem domain, so there is not much to go on. However, one metric that was proposed in c. 1990's, has been used (in a few different ways), and has a good intuiative intepretation for this problem is the Kvamme Gain statistic. This is bounded by $(-\infty,<1]$ with zero as the bounadry of random selection. The intuition is that is quantifies the percent of correctly classified positive observations relative to the region of of the observation space classified as positive.
pros of this metric are that it is bounded between 0 and ~1 for a better than random prediction, that it is easy to calculate, and that the contours of the metric are not parallel. I don't know how to explain that technically, but see image below.
cons are that it is bounded by $-\infty$ on the low end, that the contours are compressed towards (1-Specificity) instead of Sensivity, and that there are wide regions of equality even when the Sensitivity (the more important quality) is quite different. I know this is a problem with any aggregate metric, but perhaps it can be lessened
Desired Metric: I would like to have a metric that places more importance on a high sensitvity over a low specificity. Essentialy a metric that shifts the contour lines of the values so that they compress towards high sensitvity. Meaning, with an equivlant KG (i.e. KG = 0.62), it is preferable to know if this result is derived from [sens = 0.95, spec = 0.35] over the eqivalent KG derived from [sens = 0.40, spec = 0.15].
$KG = 1 - ((1 - Specificity) / Sensitivity)$
KG = 1 - (% area predicted positive / % correctly predictied positive cases)
I have explored similar metrics that are commonly used (e.g. Precision, Recall, TPR, TNR, PPV, Cohen's Kappa, etc...) and have not found a good fit in terms of behavior or intuition. I have posted grids of the Positive Likelihood Ratio (LR+) and Youden's J below for comparison
LR+ is similar to KG in that the contours are asymetrical; even more so thatn KG, but also compressed towards the wrong axis. The lack of an upper bound make cross model comparison a little tougher, but log(LR+) helps.
On the contrary, Youden's J is nicely confined between -1 and 1, but the contours are parrallel giving no weight to sensitivity.
Positive Likelihood Ratio:
$LR+ = Sensitivity / (1 - Specificty)$
Youden's J (Informedness, True Skill Statistic):
$J = Senstivity + Specificity - 1$