Searching for appropriate binary classification metric

Question

Problem:

I am working on a a binary classification problem with the following qualities:

• Highly imbalanced
• False Negative errors much more costly than False Positive
• False Positives are only False until investigated; may become True Positives
• True Negatives are by far the largest class
• Spatial in nature (raster based)

I am searching for a threshold dependent binary classification metric that utilizes the quandrant of a confusion matrix (TP,FP,TN,FN) to assess the magnitude of True Positives realtive to the area of False Positives. Preferably, this metric would use only Sensitivity and Specificity as these are the more commonly reported quantities in previous studies. Below are examples of metrics that fit some requirements, but not all.

Prime Example:

The use of classification metrics and confusion matrices is not overly common in my problem domain, so there is not much to go on. However, one metric that was proposed in c. 1990's, has been used (in a few different ways), and has a good intuiative intepretation for this problem is the Kvamme Gain statistic. This is bounded by $(-\infty,<1]$ with zero as the bounadry of random selection. The intuition is that is quantifies the percent of correctly classified positive observations relative to the region of of the observation space classified as positive.

pros of this metric are that it is bounded between 0 and ~1 for a better than random prediction, that it is easy to calculate, and that the contours of the metric are not parallel. I don't know how to explain that technically, but see image below.

cons are that it is bounded by $-\infty$ on the low end, that the contours are compressed towards (1-Specificity) instead of Sensivity, and that there are wide regions of equality even when the Sensitivity (the more important quality) is quite different. I know this is a problem with any aggregate metric, but perhaps it can be lessened

Desired Metric: I would like to have a metric that places more importance on a high sensitvity over a low specificity. Essentialy a metric that shifts the contour lines of the values so that they compress towards high sensitvity. Meaning, with an equivlant KG (i.e. KG = 0.62), it is preferable to know if this result is derived from [sens = 0.95, spec = 0.35] over the eqivalent KG derived from [sens = 0.40, spec = 0.15].

Kvamme Gain:

$KG = 1 - ((1 - Specificity) / Sensitivity)$

or

KG = 1 - (% area predicted positive / % correctly predictied positive cases)

Other Examples:

I have explored similar metrics that are commonly used (e.g. Precision, Recall, TPR, TNR, PPV, Cohen's Kappa, etc...) and have not found a good fit in terms of behavior or intuition. I have posted grids of the Positive Likelihood Ratio (LR+) and Youden's J below for comparison

LR+ is similar to KG in that the contours are asymetrical; even more so thatn KG, but also compressed towards the wrong axis. The lack of an upper bound make cross model comparison a little tougher, but log(LR+) helps.

On the contrary, Youden's J is nicely confined between -1 and 1, but the contours are parrallel giving no weight to sensitivity.

Positive Likelihood Ratio:

$LR+ = Sensitivity / (1 - Specificty)$

Youden's J (Informedness, True Skill Statistic):

$J = Senstivity + Specificity - 1$

Plots

KG

LR+

Youden's J, Informedness, TSS

Answer 1

Not exactly what you asked but MATLAB's perfcurve function uses a ratio of the costs assigned to the classes in the confusion matrix and assigns weights to them by the number of classified observations as positive and negative.

Quoting from 1:

perfcurve computes OPTROCPT for the standard ROC curve only, and sets to NaNs otherwise. To obtain the optimal operating point for the ROC curve, perfcurve first finds the slope, S, using $$S = \frac{Cost(P│N)-Cost(N│N)}{Cost(N│P)-Cost(P│P)} \cdot \frac{N}{P}$$ Cost(N|P) is the cost of misclassifying a positive class as a negative class. Cost(P|N) is the cost of misclassifying a negative class as a positive class. P = TP + FN and N = TN + FP. They are the total instance counts in the positive and negative class, respectively. perfcurve then finds the optimal operating point by moving the straight line with slope S from the upper left corner of the ROC plot (FPR = 0, TPR = 1) down and to the right, until it intersects the ROC curve.

MATLAB also gives you the possibility of assigning your own weights on every class (true positives, false positives, true negatives, false negatives).

Although this metric is intuitive, they cite no evaluation of the method.

See more here: https://ch.mathworks.com/help/stats/perfcurve.html