# Confidence Interval for accuracy - Binomial?

Imagine we fit the same data mining model on 100 different validation datasets and accuracy is always exactly 95%. If we now use the Binomial proportion confidence interval we would have a standard error of around 2%. The 95% confidence interval is therefore around (0.91,0.99) which seems quite wide for a model that always got us always 95%. Also, if we average accuracy over the 100 trials, the outcome would be the same if we would have attained 100 various accuracy measures that averaged to 0.95 (e.g. 50 times 90% and 50 times 100%). This does not seem right to me. I am also not sure if the binomial model would be right for this. Can somebody clear this up to me?
(A solution to get more narrow CI would of course be to do more repetitions, which is however not possible for me because of computing time.)

• If you always get 0.95 exactly then you are not sampling from the binomial. Commented Jan 18, 2019 at 16:30
• And if we get always around 0.95. How can I attain a CI then, (besides doing it emprically)? Could we use sth like t-distribution? Commented Jan 18, 2019 at 16:33
• What parameter is your CI intended to represent? It is extremely unclear how Binomial sampling might be involved: what is the "sample" you have in mind? In what sense is an "accuracy" value (however it might be measured) related to the counts that one observed in a Binomial model?
– whuber
Commented Jan 18, 2019 at 17:23
• It is intended to represent the „true“ accuracy measure. The sample is therefore different accuracy measures attained from e.g. fitting a model on different validation sets or by creating the model with bootstrapped samples. I also do not see the relation to Binomial dist. but I have seen people doing that (there are even (probably less professional) tutorials doing that). Thats why I am asking Commented Jan 18, 2019 at 17:40