# Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to inform users about the strength of the metrics, I'd like to include confidence intervals.

### Background on the metrics

Suppose a binary classifier that is to classify 1000 items. Of those items, 700 belong to A, and 300 to B. The results are as follows:

         Predicted
| # A | # B
-------+-----+-----
True A | 550 | 150
True B |  50 | 250


We'll call class B a positive result (1) and class A a negative one (0). So there were 550 true negatives, 150 false positives, 50 false negatives and 250 true positives.

There are some metrics defined for this classification:

$$\text{Recall} = \frac{TP}{TP+FN} = 0.833$$ $$\text{Precision} = \frac{TP}{TP + FP} = 0.625$$ $$\text{F1 score} = \frac{2}{1/recall + 1/precision} = 0.714$$

### Suggested approach

This Stack Overflow question addresses confidences of recall or precision. It suggests using an adjusted version of recall: $\text{recall} = (TP+2) / (TP+FN+4)$ and the Wilson Score interval. The final formula would be

$$p \pm Z_\alpha \cdot \text{std_error}$$

where

$$\text{std_error} = \sqrt{\frac{recall\cdot(1-recall)}{N+4}}$$

There's something I don't understand about this approach. The formula resembles the confidence interval for a binomial distribution, but it's not quite the same. Neither is it the Wilson formulation. Where does that additional $+4$ come from? Maybe it is from the recall formula.

He mentions that $p$ is calculated using the adjusted recall. That would imply $recall = \hat{p}$, which is reinforced by the error formula. So let's use the Wilson formulation for $p$. With $n=TP+FN=300$ the final calculation with a confidence of $\alpha=0.05$ yields $z=1.96$ and:

$$p \pm Z_\alpha\cdot\text{std_err} = \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm z\cdot\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n+4}} = 0.825 \pm 0.042$$

### Pondering

While this does seem like a sensible result, I still wonder about the formula and the differences between the formulae presented. What could be the basis for the $\text{std_err}$ formula? Is it better to use the Wilson formula instead?

A similar formulation could be used for calculating the confidence interval for precision using false positives instead of false negatives. How would this idea carry to the F1 score, which is a combination of the two, if at all?

My statistic skills and intuition isn't so strong yet, so any help or insight is greatly appreciated!

### Edit

Different approaches have little effect at least in this case

$$\text{Normal approach: } \hat{p} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n}} = 0.829 \pm 0.043$$ $$\text{Wilson approach: } \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n} + \frac{z^2}{4n^2}} = 0.825 \pm 0.043$$ $$\text{Above approach: } \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n+4}} = 0.825 \pm 0.042$$

For smaller samples they start to vary a bit more, especially with the interval. The Wilson approach seems to give the largest intervals.

• I couldn't find an answer to this myself, so I did it by propagating the uncertainties of the precision and the recall directly by hand. My code & explanation are here. – Cecília May 13 at 13:55