We have a series of binomial distributions, each with separate and distinct chance of success k. For some small subset of these distributions we have an absolute measure of k from the full population.
We also have a process which can look at individual trials of the binomial distributions and estimate whether they succeeded or failed. This process has an error rate.
We only have a subset of the trials from each of the distributions but we can estimate k with our process, taking our sample, determining the proportion of success to failure.
What I'm unsure of is how to derive a measure of confidence in the output of this. For the distributions where we have the known k we can calculate the percentage point difference between what we calculate, but any differences could be due to either error in the success labelling process, or the fact that we're using a sample. Over a large enough number of distributions is it possible to separate out the error of the process?
I considered using fishers exact to test the likelihood that the sample would be drawn if the two distributions were the same, but that's not quite what I'm after.
Ideally I'd like to be able to construct some confidence interval for a given distribution. "Based on the size of our sample and some estimated error in the success/failure labelling process we expect the true value to lie within this range % of the time".
|Estimated Sample Successes||Estimated Sample Failures||Population Successes||Population Failures|