This problem is somewhat involved and I have a partial solution so bear with me. I will illustrate the problem with an example. Lets say we have two processes and we want to know which has a higher success rate. So we do two sets of trials. process 1 gives 53 out of 606 and process 2 gives 32 out of 595. So the rates are

$p_1$ is 0.0538 with a 95% CL of [0.0371,0.0751]

$p_2$ is 0.0875 with a 95% CL of [0.0662,0.1128]

It would seem that $p_2$ has a higher success rate. However, both only represent one measurement of the success rate and in general these experiment are on different samples. To get the actual success rate we would need to perform multiple experiments of this type and then calculate the mean of the $p_1$s and $p_2$s. My question is how do I calculate the error on this mean? Is it calculated from the distribution of $p_1$s and $p_2$s or do I need to incorporate the confidence limit for the binomial distribution of each experiment in some way? I am happy to stay in a regime where we can make a nearly Gaussian approximation like with the numbers given above. Is this the sort of thing where one would argue from the central limit theorem that the distributions of the $p_1$s and $p_2$s are all that matter?

There is a standard way to combine several experiments if they can be assumed to have a standard error on the measurement. If the set of measurements is $a_i$ and the set of associated errors is $σ_i$, then the estimate for the true $a$ is given with accuracy $σ$ by the following:

$a = \frac{ \Sigma (a_i/σ_i^2)}{ \Sigma (1/σ_i^2)}$

$\frac{1}{σ^2} = \Sigma \frac {1}{σ_i^2}$

This just amounts to a weighted sum. The reason this does not work in general for a binomial experiment is that there is no way to calculate a σ for each measurement that makes sense. In low statistical cases like 1 success out of 4 the error is highly unsymmetrical.

However, if we are in a situation with high statistics like I gave above we can use something like the Wald interval. One can then only combine them confidently when each individual measurement is similar and derived from high statistics.

Any help is appreciated.

  • 3
    $\begingroup$ You may be over-thinking this: note that your estimate of $p_1$ is the mean of 606 observation, which is the same as averaging 303 estimates of $p_1$ of sample size 2, which is the same as averaging 101 estimates of $p_1$ of sample size 6. The CLT can be used directly on your estimates, but when $p$ is near 0 or 1 the rate of convergence is poor, and people often look to continuity corrections. $\endgroup$ – neverKnowsBest May 20 '14 at 23:04

Can you assume that the success probability among the $n_1 = 606$ or $n_2 = 595$ process realizations are equal and that the realizations are independend? If not, you'll have a hard time to do any statistics at all.

If yes --and this has to be methodologically supported by the experiment-- you can model each realization $X_{ik}$ as a Bernoulli experiment with success probability $p_i$. In other terms, you can consider $X_{ik}$ as a random variable with $P(X_{ik} = 1) = p_i$ and $P(X_{ik} = 0) = 1-p_i$.

Now you observe that $Var(X_{ik}) = p_i(1-p_i)<\infty$, so as your realizations are independent, you can apply the central limit theorem on $\frac{1}{n_i}\sum_{k=1}^{n_i}X_{ik}=\hat{p_i}$.

As you wished, this boils your situation down to calculating a (Wald-)confidence interval for $p_1 - p_2$: The variance $Var(\hat{p_1} - \hat{p_2})=\sum_{i=1}^2\frac{p_i}{n_i}(1-p_i)$ can be estimated by plugging in $\hat{p_i}$ for $p_i$. So with $z \approx 1.96$ you get $$\hat{p_1} - \hat{p_2} \pm z \sqrt{\sum_{i=1}^2\frac{\hat{p}_i}{n_i}(1-\hat{p}_i)}$$ as confidence bounds for the difference.

Now if you have many such success rates and want to know which is best among all of them, you'll need simultaneous confidence intervals. (I'll elaborate on this if you need.)

  • $\begingroup$ We can assume all of the Bernoulli trials are independent and that there are only two underlying rates (p1 and p2). A typical example would be testing which medical treatment gives a better recovery rate. I think the comment on the original post solves the multiple trial issue by saying that you can use all events at once by simply merging the experiments together. Do you agree that this is safe? The question now is how do you devise a one tailed or two tailed test to compare the two processes. What you give would be for a Gaussian assumption. $\endgroup$ – Keith May 22 '15 at 23:05
  • $\begingroup$ This may be the answer stats.stackexchange.com/questions/113602/… $\endgroup$ – Keith May 22 '15 at 23:35
  • $\begingroup$ Andrew tells you how to calculate the confidence intervals: stats.stackexchange.com/questions/25299/…. This is best for you. There is nothing wrong in your case with assuming asymptotically gaussian distribution. In fact Fisher's exact test might be too expensive to calculate. $\endgroup$ – Horst Grünbusch May 23 '15 at 11:06
  • $\begingroup$ If you really want to know which trial reveals a better success rate than all the other trials, there is no other way than multiple testing/simultaneous CI for pairwise comparisons. I don't see how you would need to merge the data unless you are just want to know if all success are not equal (which is less informative, of course, as you would not identify the best trial). I think the comment of @neverKnowsBest is just to point at how simple variance (estimation) of Bernoulli rv is. $\endgroup$ – Horst Grünbusch May 23 '15 at 11:17
  • $\begingroup$ Maybe I did not explain properly. There are only two processes being tested. Each experiment consists of multiple trails so each experiment is binomial. For each process we then have multiple experiments each of which consists of multiple trials. What I was getting at is that we can combine the experiments for each precess by simply adding the successes and failures. These sums can then be used in a Fisher's exact test. $\endgroup$ – Keith May 25 '15 at 18:08

In the case where you have the number of successes and the number of trials for each experiment, it can be solved exactly. This is almost always the case in a situation such as this.

You can combine all of the experiments for each process so that you only have two binomial distributions to compare. This would be done by a simple sum over all the experiments to get the totals for each process.

$\frac{Sucesses_{tot}}{n_{tot}} = \frac {\Sigma Sucesses_exp}{\Sigma n_exp}$

You would then have the total number of successes and total number of trails for each process. You can then compare your two processes with Fisher's Exact Test


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.