Test if the difference in sampled success rate of Bernoulli distributions are significant

Question

I am drawing samples from two distinct Bernoulli distributions $Y_1$, $Y_2$.

Suppose after $n$ trials each, I find values for $\check p_1$ and $\check p_2$, the sample success probabilities, for $Y_1$, $Y_2$ respectively.

How do I test if the difference of $\check p_1$ and $\check p_2$ is significant with a given confidence (say 95% p = 0.05?)

I.e. How do I check that, given my samples over $n$ trials, the true value of $p_1$ > $p_2$

Answer 1

I'm going to assume you meant $n$ trials each, just for simplicity. Also, I'm going to assume that the sample success probabilities are computed as averages of your $n$ samples from each distribution.

I'll denote the true probability of success associated with $Y_1$ and $Y_2$ as $p_1$ and $p_2$, respectively. Also, define $\hat{P}_1$ and $\hat{P}_2$ which are the random variables representing our sample proportions, as opposed to $\hat{p}_1$ and $\hat{p}_2$ which are the observed sample proportions. By the Central Limit Theorem we get

$$\frac{\sqrt{n}(\hat{P}_1 - p_1)}{\sqrt{\hat{p}_1(1-\hat{p}_1)}} \xrightarrow{d} \mathcal{N}(0,1)$$

and

$$\frac{\sqrt{n}(\hat{P}_2 - p_2)}{\sqrt{\hat{p}_2(1-\hat{p}_2)}} \xrightarrow{d} \mathcal{N}(0,1)$$.

So for $n$ large, we can approximate the distributions of each of our statistics. Specifically, $\hat{P}_1 \sim \mathcal{N}\left(p_1, \frac{\hat{p}_1(1-\hat{p}_1)}{n}\right)$ and $\hat{P}_2 \sim \mathcal{N}\left(p_2, \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right)$.

We finally get to our test. Our hypotheses are

$$H_0: p_1 - p_2 = 0$$ $$H_1: p_1 - p_2 > 0.$$

Now, we assume we're operating under the null hypothesis, so $p_1 - p_2 = 0$. Under this assumption, we know, approximately,

$$\hat{P}_1 - \hat{P}_2 \sim \mathcal{N}\left(0, \frac{\hat{p}_1(1-\hat{p}_1)}{n} + \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right).$$

We know the distribution of the difference between sample proportions under $H_0$. Given our data, we also can calculate our "observed" difference in sample proportions, $\hat{p}_1 - \hat{p}_2$.

If $P(\hat{P}_1 - \hat{P}_2 > \hat{p}_1 - \hat{p}_2) \leq 0.05$, then we reject $H_0$. Or else, we fail to reject $H_0$.

The interpretation is, if we assume we're operating under the null and we observe a value ($\hat{p}_1 - \hat{p}_2$) that is "inconsistent enough" with the null hypothesis, then we're going to reject $H_0$! This level is precisely $0.05$, because we specified the size of our test.

Answer 2

Suppose you have $X_1$ "successes" in $n_1$ Bernoulli trials for Population 1 and $X_2$ successes in $n_2$ Bernoulli trials for Population 2. Also suppose that $n_1$ and $n_2$ are large enough that the respective estimates $\hat p_i$ of the Success probabilities $p_i, i = 1,2$ can be assumed to be approximately normally distributed. Then you can test $H_0: p_1 = p_2$ against $H_a: p_i \ne p_2$ in R by using the procedure prop.test.

In particular, suppose that in a vaccine trial, you $X_1 = 55$ out 0f $492$ subjects in the Control group (receiving placebo injections) have become infected with the disease the vaccine is intended to prevent. Also, suppose that in the treatment group (vaccine injections) $X_2 = 12$ out of $502$ became infected. Then prop.test gives the following results:

prop.test(c(55, 12), c(492, 502), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(55, 12) out of c(492, 502)
X-squared = 30.53, df = 1, p-value = 3.288e-08
alternative hypothesis: two.sided
95 percent confidence interval:
 0.05700047 0.11876800
sample estimates:
    prop 1     prop 2 
0.11178862 0.02390438 

The 'continuity correction' (not needed because of the relatively large sample sizes) has been suppressed by using the parameter cor=F. The P-value (essentially $0)$ indicates you can reject $H_0$ at the 5% level of significance (or lower levels such as 0.1%).

Notes: This test in R implements a test described by NIST. One-sided tests use the parameter alt="less" or alt="greater' (depending on direction). [It is essentially the same test discussed in another Answer to your question.] For smaller sample sizes consider using Fisher's Exact test.