I am drawing samples from two distinct Bernoulli distributions $Y_1$, $Y_2$.

Suppose after $n$ trials each, I find values for $\check p_1$ and $\check p_2$, the sample success probabilities, for $Y_1$, $Y_2$ respectively.

How do I test if the difference of $\check p_1$ and $\check p_2$ is significant with a given confidence (say 95% p = 0.05?)

I.e. How do I check that, given my samples over $n$ trials, the true value of $p_1$ > $p_2$


I'm going to assume you meant $n$ trials each, just for simplicity. Also, I'm going to assume that the sample success probabilities are computed as averages of your $n$ samples from each distribution.

I'll denote the true probability of success associated with $Y_1$ and $Y_2$ as $p_1$ and $p_2$, respectively. Also, define $\hat{P}_1$ and $\hat{P}_2$ which are the random variables representing our sample proportions, as opposed to $\hat{p}_1$ and $\hat{p}_2$ which are the observed sample proportions. By the Central Limit Theorem we get

$$\frac{\sqrt{n}(\hat{P}_1 - p_1)}{\sqrt{\hat{p}_1(1-\hat{p}_1)}} \xrightarrow{d} \mathcal{N}(0,1)$$


$$\frac{\sqrt{n}(\hat{P}_2 - p_2)}{\sqrt{\hat{p}_2(1-\hat{p}_2)}} \xrightarrow{d} \mathcal{N}(0,1)$$.

So for $n$ large, we can approximate the distributions of each of our statistics. Specifically, $\hat{P}_1 \sim \mathcal{N}\left(p_1, \frac{\hat{p}_1(1-\hat{p}_1)}{n}\right)$ and $\hat{P}_2 \sim \mathcal{N}\left(p_2, \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right)$.

We finally get to our test. Our hypotheses are

$$H_0: p_1 - p_2 = 0$$ $$H_1: p_1 - p_2 > 0.$$

Now, we assume we're operating under the null hypothesis, so $p_1 - p_2 = 0$. Under this assumption, we know, approximately,

$$\hat{P}_1 - \hat{P}_2 \sim \mathcal{N}\left(0, \frac{\hat{p}_1(1-\hat{p}_1)}{n} + \frac{\hat{p}_2(1-\hat{p}_2)}{n}\right).$$

We know the distribution of the difference between sample proportions under $H_0$. Given our data, we also can calculate our "observed" difference in sample proportions, $\hat{p}_1 - \hat{p}_2$.

If $P(\hat{P}_1 - \hat{P}_2 > \hat{p}_1 - \hat{p}_2) \leq 0.05$, then we reject $H_0$. Or else, we fail to reject $H_0$.

The interpretation is, if we assume we're operating under the null and we observe a value ($\hat{p}_1 - \hat{p}_2$) that is "inconsistent enough" with the null hypothesis, then we're going to reject $H_0$! This level is precisely $0.05$, because we specified the size of our test.


Suppose you have $X_1$ "successes" in $n_1$ Bernoulli trials for Population 1 and $X_2$ successes in $n_2$ Bernoulli trials for Population 2. Also suppose that $n_1$ and $n_2$ are large enough that the respective estimates $\hat p_i$ of the Success probabilities $p_i, i = 1,2$ can be assumed to be approximately normally distributed. Then you can test $H_0: p_1 = p_2$ against $H_a: p_i \ne p_2$ in R by using the procedure prop.test.

In particular, suppose that in a vaccine trial, you $X_1 = 55$ out 0f $492$ subjects in the Control group (receiving placebo injections) have become infected with the disease the vaccine is intended to prevent. Also, suppose that in the treatment group (vaccine injections) $X_2 = 12$ out of $502$ became infected. Then prop.test gives the following results:

prop.test(c(55, 12), c(492, 502), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(55, 12) out of c(492, 502)
X-squared = 30.53, df = 1, p-value = 3.288e-08
alternative hypothesis: two.sided
95 percent confidence interval:
 0.05700047 0.11876800
sample estimates:
    prop 1     prop 2 
0.11178862 0.02390438 

The 'continuity correction' (not needed because of the relatively large sample sizes) has been suppressed by using the parameter cor=F. The P-value (essentially $0)$ indicates you can reject $H_0$ at the 5% level of significance (or lower levels such as 0.1%).

Notes: This test in R implements a test described by NIST. One-sided tests use the parameter alt="less" or alt="greater' (depending on direction). [It is essentially the same test discussed in another Answer to your question.] For smaller sample sizes consider using Fisher's Exact test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.