# How do I calculate the uncertainty in process which estimates binomial distributions and has its own error rate

We have a series of binomial distributions, each with separate and distinct chance of success k. For some small subset of these distributions we have an absolute measure of k from the full population.

We also have a process which can look at individual trials of the binomial distributions and estimate whether they succeeded or failed. This process has an error rate.

We only have a subset of the trials from each of the distributions but we can estimate k with our process, taking our sample, determining the proportion of success to failure.

What I'm unsure of is how to derive a measure of confidence in the output of this. For the distributions where we have the known k we can calculate the percentage point difference between what we calculate, but any differences could be due to either error in the success labelling process, or the fact that we're using a sample. Over a large enough number of distributions is it possible to separate out the error of the process?

I considered using fishers exact to test the likelihood that the sample would be drawn if the two distributions were the same, but that's not quite what I'm after.

Ideally I'd like to be able to construct some confidence interval for a given distribution. "Based on the size of our sample and some estimated error in the success/failure labelling process we expect the true value to lie within this range % of the time".

example results:

Estimated Sample Successes Estimated Sample Failures Population Successes Population Failures
22 80 336 892
8 36 226 525
6 13 100 175
1 9 11 137

A number of confidence intervals are in use for the probability of success, which vary in the complexity of their computation and in their coverage probability $$P_{cov}$$ (which depends on the unknown true probability $$p$$).

The interval with $$P_{cov}$$ guaranteed to be greater than the nominal level $$1-\alpha$$ for all possible values of the true probability $$p$$ is known as the "Clopper-Pearson interval" or the "exact interval". In many cases, its coverage probability is considerably greater then $$1-\alpha$$, though. Therefore other intervals are in use that have $$P_{cov}<1-\alpha$$ for some values of $$p$$, but deviate not too much. Brown et al. (2001) found the "Wilson interval" to be a good compromise, which has the nice property of being computable through a closed formula. In my own experiments (second reference given below), I found the "Bayesian Highest Posterior Density interval" to have better coverage probability than the Wilson interval for true values $$p$$ close to one or zero (typical for estimating error/recognition rates), and this despite its smaller width. Its downside is that it is much more difficult to compute (no closed formula, but a computer algorithm).

The R package binom implements many confidence intervals (albeit not the Bayesian HPD interval) in the function binom.confint:

> library(binom)
> k <- 1
> n <- 10
> binom.confint(k,n, conf.level=0.95, methods=c("wilson","exact"))
method x  n mean       lower     upper
1  exact 1 10  0.1 0.002528579 0.4450161
2 wilson 1 10  0.1 0.017876213 0.4041500

The HPD interval can be computed with the R package HDInterval as follows:

> library(HDInterval)
> k <- 1
> n <- 10
> hdi(qbeta, 0.95, shape1=(k+1), shape2=(n-k+1))
lower     upper
0.006301509 0.367513183

Brown, Cai, DasGupta: "Interval estimation for a binomial proportion." Statistical science, vol. 16, no. 2, pp. 101–117, 2001

Dalitz: "Construction of confidence intervals." Technical Report No. 2017-01, pp. 15-28, Hochschule Niederrhein, Fachbereich Elektrotechnik und Informatik, 2017

• Thanks for the detailed answer, it iss helpful but there is a complication. Your answer assumes that we can accurately measure the successes in our sample, but the problem I have is that the process to do this has it's own unknown rate of error. My thinking (based on your answer) is that with enough examples I can construct 95% coverage intervals, and then measure the rate at which the corresponding known populations fall within the sample CI. If there is non-negligible error in the success labeller than the actual number of populations outside of the CI will be greater than 5% Dec 13, 2022 at 14:22
• If you can estimate the probability that a measurement of "success" is correct, you can try to estimate the uncertainty in the probability estimate $p$ by means of a Monte Carlo simulation which uses the estimated parameters as the model parameters for the uncertainty and $p$.. Dec 14, 2022 at 15:21