How do I calculate the uncertainty in process which estimates binomial distributions and has its own error rate

Question

We have a series of binomial distributions, each with separate and distinct chance of success k. For some small subset of these distributions we have an absolute measure of k from the full population.

We also have a process which can look at individual trials of the binomial distributions and estimate whether they succeeded or failed. This process has an error rate.

We only have a subset of the trials from each of the distributions but we can estimate k with our process, taking our sample, determining the proportion of success to failure.

What I'm unsure of is how to derive a measure of confidence in the output of this. For the distributions where we have the known k we can calculate the percentage point difference between what we calculate, but any differences could be due to either error in the success labelling process, or the fact that we're using a sample. Over a large enough number of distributions is it possible to separate out the error of the process?

I considered using fishers exact to test the likelihood that the sample would be drawn if the two distributions were the same, but that's not quite what I'm after.

Ideally I'd like to be able to construct some confidence interval for a given distribution. "Based on the size of our sample and some estimated error in the success/failure labelling process we expect the true value to lie within this range % of the time".

example results:

Estimated Sample Successes	Estimated Sample Failures	Population Successes	Population Failures
22	80	336	892
8	36	226	525
6	13	100	175
1	9	11	137

Answer 1

A number of confidence intervals are in use for the probability of success, which vary in the complexity of their computation and in their coverage probability $P_{cov}$ (which depends on the unknown true probability $p$).

The interval with $P_{cov}$ guaranteed to be greater than the nominal level $1-\alpha$ for all possible values of the true probability $p$ is known as the "Clopper-Pearson interval" or the "exact interval". In many cases, its coverage probability is considerably greater then $1-\alpha$, though. Therefore other intervals are in use that have $P_{cov}<1-\alpha$ for some values of $p$, but deviate not too much. Brown et al. (2001) found the "Wilson interval" to be a good compromise, which has the nice property of being computable through a closed formula. In my own experiments (second reference given below), I found the "Bayesian Highest Posterior Density interval" to have better coverage probability than the Wilson interval for true values $p$ close to one or zero (typical for estimating error/recognition rates), and this despite its smaller width. Its downside is that it is much more difficult to compute (no closed formula, but a computer algorithm).

The R package binom implements many confidence intervals (albeit not the Bayesian HPD interval) in the function binom.confint:

> library(binom)
> k <- 1
> n <- 10
> binom.confint(k,n, conf.level=0.95, methods=c("wilson","exact"))
  method x  n mean       lower     upper
1  exact 1 10  0.1 0.002528579 0.4450161
2 wilson 1 10  0.1 0.017876213 0.4041500

The HPD interval can be computed with the R package HDInterval as follows:

> library(HDInterval)
> k <- 1
> n <- 10
> hdi(qbeta, 0.95, shape1=(k+1), shape2=(n-k+1))
        lower     upper 
0.006301509 0.367513183

Brown, Cai, DasGupta: "Interval estimation for a binomial proportion." Statistical science, vol. 16, no. 2, pp. 101–117, 2001

Dalitz: "Construction of confidence intervals." Technical Report No. 2017-01, pp. 15-28, Hochschule Niederrhein, Fachbereich Elektrotechnik und Informatik, 2017