0
$\begingroup$

I have an imbalanced data with binary label where there are only 4% positive labels among all examples. I want to evaluate my model on the dataset, and I wonder what is the best way (best metric) to evaluate it.

My model produces predicted probabilities in $[0, 1]$ and it gives prediction in $\{0, 1\}$ by thresholding, and 0.5 is not the best threshold since the label is imbalanced. When I plot acc, precision, recall, and f1 score with thresholds from 0.01 to 0.99, I got the following graph:

enter image description here

For me, recall (sensitivity) is the most important metric. However, I can make it very high (>0.95) by simply set threshold as small as possible, which make the model to predict almost every example as negative. (Obviously, this makes precision super low). I can use f1, AUROC, or AUPRC, which mitigates such thresholding issues, but I wonder if there's a better way to evaluate recall in a reasonable way. Thanks in advance.

$\endgroup$

1 Answer 1

1
$\begingroup$

I actually think that AUPRC is a good way to go -- it essentially measures precision as a function of recall at varying thresholds -- but since you've mentioned that already, there's one more thing you can consider -- the F-beta measure.

This builds directly off of F1-score. As you know, F1 is given by

$$F_1 = \frac{2P\cdot R}{P+R}$$

where $P$ is precision and $R$ is recall. What if you want to control the "balance" between precision and recall in this metric? That's where the F-beta measure comes in, which takes a positive scalar parameter $\beta$ as follows:

$$F_\beta = (1 + \beta^2) \frac{P \cdot R}{\beta^2 \cdot P + R}.$$

If you want to make recall matter more, make $\beta$ larger. $\beta=2$ is usually a good start, though this will need to be tuned experimentally. This should be sufficient for your purposes; I think it's important as well to remember that metrics are heavily task-dependent (which you see to be thinking about), so it's okay to try a few different metrics to see which one communicates your "point" most effectively.

If you're curious about why this metric weighs recall/precision more heavily depending on the setting of $\beta$, I've included an explanation below.

Extra Details (Why Does This Work)

Simplifying in terms of false/true positives/negatives, we can rewrite this as $$F_\beta = (1 + \beta^2) \frac{\frac{TP}{TP + FP} \cdot \frac{TP}{TP + FN}}{\beta^2 \cdot \frac{TP}{TP + FP} +\frac{TP}{TP + FN}}$$ $$= (1 + \beta^2) \frac{\frac{TP}{(TP + FP)(TP + FN)}}{\beta^2 \cdot \frac{1}{TP + FP} +\frac{1}{TP + FN}}$$ $$= (1 + \beta^2) \frac{\frac{TP}{(TP + FP)(TP + FN)}}{\beta^2 \cdot \frac{TP + FN}{(TP + FP)(TP + FN)} +\frac{TP + FP}{(TP + FP)(TP + FN)}}$$ $$= \frac{(1 + \beta^2)\cdot TP}{\beta^2 (TP + FN) +TP + FP}$$ $$= \frac{(1 + \beta^2)\cdot TP}{(1 + \beta^2) \cdot TP + \beta^2 \cdot FN + FP}.$$

Thus, you can observe the influence of $\beta^2$ as a "weighing" term in the denominator of this metric. Specifically, the $FN$ term is multiplied by $\beta^2$; hence, increasing $\beta$ increases the influence/penalty that Type II Errors incur -- which is consistent with recall being weighted higher. Conversely, you can see how decreasing $\beta^2$ would result in a metric that puts more importance on precision. How $\beta$ corresponds to weighing recall $\beta$ times more heavily than precision is further explorer in this answer

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.