# Precision and Recall for highly-imbalanced data

I have an imbalanced data with binary label where there are only 4% positive labels among all examples. I want to evaluate my model on the dataset, and I wonder what is the best way (best metric) to evaluate it.

My model produces predicted probabilities in $$[0, 1]$$ and it gives prediction in $$\{0, 1\}$$ by thresholding, and 0.5 is not the best threshold since the label is imbalanced. When I plot acc, precision, recall, and f1 score with thresholds from 0.01 to 0.99, I got the following graph:

For me, recall (sensitivity) is the most important metric. However, I can make it very high (>0.95) by simply set threshold as small as possible, which make the model to predict almost every example as negative. (Obviously, this makes precision super low). I can use f1, AUROC, or AUPRC, which mitigates such thresholding issues, but I wonder if there's a better way to evaluate recall in a reasonable way. Thanks in advance.

I actually think that AUPRC is a good way to go -- it essentially measures precision as a function of recall at varying thresholds -- but since you've mentioned that already, there's one more thing you can consider -- the F-beta measure.

This builds directly off of F1-score. As you know, F1 is given by

$$F_1 = \frac{2P\cdot R}{P+R}$$

where $$P$$ is precision and $$R$$ is recall. What if you want to control the "balance" between precision and recall in this metric? That's where the F-beta measure comes in, which takes a positive scalar parameter $$\beta$$ as follows:

$$F_\beta = (1 + \beta^2) \frac{P \cdot R}{\beta^2 \cdot P + R}.$$

If you want to make recall matter more, make $$\beta$$ larger. $$\beta=2$$ is usually a good start, though this will need to be tuned experimentally. This should be sufficient for your purposes; I think it's important as well to remember that metrics are heavily task-dependent (which you see to be thinking about), so it's okay to try a few different metrics to see which one communicates your "point" most effectively.

If you're curious about why this metric weighs recall/precision more heavily depending on the setting of $$\beta$$, I've included an explanation below.

### Extra Details (Why Does This Work)

Simplifying in terms of false/true positives/negatives, we can rewrite this as $$F_\beta = (1 + \beta^2) \frac{\frac{TP}{TP + FP} \cdot \frac{TP}{TP + FN}}{\beta^2 \cdot \frac{TP}{TP + FP} +\frac{TP}{TP + FN}}$$ $$= (1 + \beta^2) \frac{\frac{TP}{(TP + FP)(TP + FN)}}{\beta^2 \cdot \frac{1}{TP + FP} +\frac{1}{TP + FN}}$$ $$= (1 + \beta^2) \frac{\frac{TP}{(TP + FP)(TP + FN)}}{\beta^2 \cdot \frac{TP + FN}{(TP + FP)(TP + FN)} +\frac{TP + FP}{(TP + FP)(TP + FN)}}$$ $$= \frac{(1 + \beta^2)\cdot TP}{\beta^2 (TP + FN) +TP + FP}$$ $$= \frac{(1 + \beta^2)\cdot TP}{(1 + \beta^2) \cdot TP + \beta^2 \cdot FN + FP}.$$

Thus, you can observe the influence of $$\beta^2$$ as a "weighing" term in the denominator of this metric. Specifically, the $$FN$$ term is multiplied by $$\beta^2$$; hence, increasing $$\beta$$ increases the influence/penalty that Type II Errors incur -- which is consistent with recall being weighted higher. Conversely, you can see how decreasing $$\beta^2$$ would result in a metric that puts more importance on precision. How $$\beta$$ corresponds to weighing recall $$\beta$$ times more heavily than precision is further explorer in this answer