# Test if the difference in sampled success rate of Bernoulli distributions are significant

I am drawing samples from two distinct Bernoulli distributions $$Y_1$$, $$Y_2$$.

Suppose after $$n$$ trials each, I find values for $$\check p_1$$ and $$\check p_2$$, the sample success probabilities, for $$Y_1$$, $$Y_2$$ respectively.

How do I test if the difference of $$\check p_1$$ and $$\check p_2$$ is significant with a given confidence (say 95% p = 0.05?)

I.e. How do I check that, given my samples over $$n$$ trials, the true value of $$p_1$$ > $$p_2$$

I'm going to assume you meant $$n$$ trials each, just for simplicity. Also, I'm going to assume that the sample success probabilities are computed as averages of your $$n$$ samples from each distribution.

I'll denote the true probability of success associated with $$Y_1$$ and $$Y_2$$ as $$p_1$$ and $$p_2$$, respectively. Also, define $$\hat{P}_1$$ and $$\hat{P}_2$$ which are the random variables representing our sample proportions, as opposed to $$\hat{p}_1$$ and $$\hat{p}_2$$ which are the observed sample proportions. By the Central Limit Theorem we get

$$\frac{\sqrt{n}(\hat{P}_1 - p_1)}{\sqrt{\hat{p}_1(1-\hat{p}_1)}} \xrightarrow{d} \mathcal{N}(0,1)$$

and

$$\frac{\sqrt{n}(\hat{P}_2 - p_2)}{\sqrt{\hat{p}_2(1-\hat{p}_2)}} \xrightarrow{d} \mathcal{N}(0,1)$$.

So for $$n$$ large, we can approximate the distributions of each of our statistics. Specifically, $$\hat{P}_1 \sim \mathcal{N}\left(p_1, \frac{\hat{p}_1(1-\hat{p}_1)}{n}\right)$$ and $$\hat{P}_2 \sim \mathcal{N}\left(p_2, \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right)$$.

We finally get to our test. Our hypotheses are

$$H_0: p_1 - p_2 = 0$$ $$H_1: p_1 - p_2 > 0.$$

Now, we assume we're operating under the null hypothesis, so $$p_1 - p_2 = 0$$. Under this assumption, we know, approximately,

$$\hat{P}_1 - \hat{P}_2 \sim \mathcal{N}\left(0, \frac{\hat{p}_1(1-\hat{p}_1)}{n} + \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right).$$

We know the distribution of the difference between sample proportions under $$H_0$$. Given our data, we also can calculate our "observed" difference in sample proportions, $$\hat{p}_1 - \hat{p}_2$$.

If $$P(\hat{P}_1 - \hat{P}_2 > \hat{p}_1 - \hat{p}_2) \leq 0.05$$, then we reject $$H_0$$. Or else, we fail to reject $$H_0$$.

The interpretation is, if we assume we're operating under the null and we observe a value ($$\hat{p}_1 - \hat{p}_2$$) that is "inconsistent enough" with the null hypothesis, then we're going to reject $$H_0$$! This level is precisely $$0.05$$, because we specified the size of our test.

Suppose you have $$X_1$$ "successes" in $$n_1$$ Bernoulli trials for Population 1 and $$X_2$$ successes in $$n_2$$ Bernoulli trials for Population 2. Also suppose that $$n_1$$ and $$n_2$$ are large enough that the respective estimates $$\hat p_i$$ of the Success probabilities $$p_i, i = 1,2$$ can be assumed to be approximately normally distributed. Then you can test $$H_0: p_1 = p_2$$ against $$H_a: p_i \ne p_2$$ in R by using the procedure prop.test.

In particular, suppose that in a vaccine trial, you $$X_1 = 55$$ out 0f $$492$$ subjects in the Control group (receiving placebo injections) have become infected with the disease the vaccine is intended to prevent. Also, suppose that in the treatment group (vaccine injections) $$X_2 = 12$$ out of $$502$$ became infected. Then prop.test gives the following results:

prop.test(c(55, 12), c(492, 502), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(55, 12) out of c(492, 502)
X-squared = 30.53, df = 1, p-value = 3.288e-08
alternative hypothesis: two.sided
95 percent confidence interval:
0.05700047 0.11876800
sample estimates:
prop 1     prop 2
0.11178862 0.02390438


The 'continuity correction' (not needed because of the relatively large sample sizes) has been suppressed by using the parameter cor=F. The P-value (essentially $$0)$$ indicates you can reject $$H_0$$ at the 5% level of significance (or lower levels such as 0.1%).

Notes: This test in R implements a test described by NIST. One-sided tests use the parameter alt="less" or alt="greater' (depending on direction). [It is essentially the same test discussed in another Answer to your question.] For smaller sample sizes consider using Fisher's Exact test.