0
$\begingroup$

I have a dataset with billions of values that either belong to a certain group or not. Let's say for example that I have sports equipment and I'm categorizing as 'tennis' or some other sport. I sample 100 products at random and manually determine whether my model has categorized within the correct category. I find that 95 are correctly categorized.

I then go to my stakeholders and say that my model is 95% accurate. However, I think there is missing nuance. I should probably say something like, "I'm 99% confident that the model is between 94% and 96% accurate" -- not something usually mentioned along with an accuracy metric.

Multiple questions: 1) How do I state my confidence that I'm 95%+ accurate 2) How do I determine my range and state my confidence there 3) Is my experiment sampling 100 items giving me a different metric than I think it is.

I know that standard error (sample std. dev/sqrt(n sampled)) will give me the normal distribution of the error on a metric (e.g. weather temperature). Does this also hold where the metric is an accuracy measurement (i.e. another metric derived based on the data)? And in this case is the n samples the number of accuracy measurements (i.e. I sampled 100 n times) or is n related to the number of samples I took from the population for my accuracy measurement?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You're on the right track with standard errors, but note that the standard error you mention (sample std. dev/sqrt(n sampled)) is the standard error of one particular parameter, the mean. The standard error of the accuracy proportion will capture the sample-to-sample variability of the accuracy estimate in the way you're looking for, but requires a different formula. $\endgroup$
    – Nuclear Hoagie
    Commented Jul 21, 2022 at 20:10

1 Answer 1

Reset to default
2
$\begingroup$

Assuming appropriate independence, you're looking at estimating a proportion from Bernoulli draws. The standard error of this estimation is

$$\mathrm{se}_p = \sqrt{\frac{p(1-p)}{n-1}}$$

where $n$ is the number of samples you drew to estimate $p$.

In your case, this means the standard error of your 95 % is

$$\mathrm{se}_p = \sqrt{\frac{0.95(0.05)}{99}} = 0.022$$.

You can then be 90 % confident that the population proportion is in the range $p\pm1.645\mathrm{se}_p$, which is the range between 91 % and 99 %.

If you want to be 99 % sure, the appropriate two-sided Z score is around 2.48, meaning the confidence interval is between 89 % and 100 %.

See e.g. chapter 3 of Cochran's Sampling Techniques (1953) for more.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.