# Find the confidence in an accuracy measurement

I have a dataset with billions of values that either belong to a certain group or not. Let's say for example that I have sports equipment and I'm categorizing as 'tennis' or some other sport. I sample 100 products at random and manually determine whether my model has categorized within the correct category. I find that 95 are correctly categorized.

I then go to my stakeholders and say that my model is 95% accurate. However, I think there is missing nuance. I should probably say something like, "I'm 99% confident that the model is between 94% and 96% accurate" -- not something usually mentioned along with an accuracy metric.

Multiple questions: 1) How do I state my confidence that I'm 95%+ accurate 2) How do I determine my range and state my confidence there 3) Is my experiment sampling 100 items giving me a different metric than I think it is.

I know that standard error (sample std. dev/sqrt(n sampled)) will give me the normal distribution of the error on a metric (e.g. weather temperature). Does this also hold where the metric is an accuracy measurement (i.e. another metric derived based on the data)? And in this case is the n samples the number of accuracy measurements (i.e. I sampled 100 n times) or is n related to the number of samples I took from the population for my accuracy measurement?

• You're on the right track with standard errors, but note that the standard error you mention (sample std. dev/sqrt(n sampled)) is the standard error of one particular parameter, the mean. The standard error of the accuracy proportion will capture the sample-to-sample variability of the accuracy estimate in the way you're looking for, but requires a different formula. Commented Jul 21, 2022 at 20:10

Assuming appropriate independence, you're looking at estimating a proportion from Bernoulli draws. The standard error of this estimation is

$$\mathrm{se}_p = \sqrt{\frac{p(1-p)}{n-1}}$$

where $$n$$ is the number of samples you drew to estimate $$p$$.

In your case, this means the standard error of your 95 % is

$$\mathrm{se}_p = \sqrt{\frac{0.95(0.05)}{99}} = 0.022$$.

You can then be 90 % confident that the population proportion is in the range $$p\pm1.645\mathrm{se}_p$$, which is the range between 91 % and 99 %.

If you want to be 99 % sure, the appropriate two-sided Z score is around 2.48, meaning the confidence interval is between 89 % and 100 %.

See e.g. chapter 3 of Cochran's Sampling Techniques (1953) for more.