5
$\begingroup$

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to inform users about the strength of the metrics, I'd like to include confidence intervals.

Background on the metrics

Suppose a binary classifier that is to classify 1000 items. Of those items, 700 belong to A, and 300 to B. The results are as follows:

         Predicted
       | # A | # B
-------+-----+-----
True A | 550 | 150
True B |  50 | 250

We'll call class B a positive result (1) and class A a negative one (0). So there were 550 true negatives, 150 false positives, 50 false negatives and 250 true positives.

There are some metrics defined for this classification:

$$\text{Recall} = \frac{TP}{TP+FN} = 0.833$$ $$\text{Precision} = \frac{TP}{TP + FP} = 0.625$$ $$\text{F1 score} = \frac{2}{1/recall + 1/precision} = 0.714$$

Suggested approach

This Stack Overflow question addresses confidences of recall or precision. It suggests using an adjusted version of recall: $\text{recall} = (TP+2) / (TP+FN+4)$ and the Wilson Score interval. The final formula would be

$$p \pm Z_\alpha \cdot \text{std_error}$$

where

$$\text{std_error} = \sqrt{\frac{recall\cdot(1-recall)}{N+4}}$$

There's something I don't understand about this approach. The formula resembles the confidence interval for a binomial distribution, but it's not quite the same. Neither is it the Wilson formulation. Where does that additional $+4$ come from? Maybe it is from the recall formula.

He mentions that $p$ is calculated using the adjusted recall. That would imply $recall = \hat{p}$, which is reinforced by the error formula. So let's use the Wilson formulation for $p$. With $n=TP+FN=300$ the final calculation with a confidence of $\alpha=0.05$ yields $z=1.96$ and:

$$p \pm Z_\alpha\cdot\text{std_err} = \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm z\cdot\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n+4}} = 0.825 \pm 0.042$$

Pondering

While this does seem like a sensible result, I still wonder about the formula and the differences between the formulae presented. What could be the basis for the $\text{std_err}$ formula? Is it better to use the Wilson formula instead?

A similar formulation could be used for calculating the confidence interval for precision using false positives instead of false negatives. How would this idea carry to the F1 score, which is a combination of the two, if at all?

My statistic skills and intuition isn't so strong yet, so any help or insight is greatly appreciated!


Edit

Different approaches have little effect at least in this case

$$\text{Normal approach: } \hat{p} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n}} = 0.829 \pm 0.043$$ $$\text{Wilson approach: } \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n} + \frac{z^2}{4n^2}} = 0.825 \pm 0.043$$ $$\text{Above approach: } \frac{\hat{p} + z^2/(2n)}{1+z^2/n} \pm Z_\alpha\sqrt{\frac{\hat{p}\cdot(1-\hat{p})}{n+4}} = 0.825 \pm 0.042$$

For smaller samples they start to vary a bit more, especially with the interval. The Wilson approach seems to give the largest intervals.

$\endgroup$
  • $\begingroup$ I couldn't find an answer to this myself, so I did it by propagating the uncertainties of the precision and the recall directly by hand. My code & explanation are here. $\endgroup$ – Cecília May 13 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.