Suppose that we are interested in comparing two approximately normal sampling distributions described by random variables $ \displaystyle \frac{Y_1}{n_1} = N(p_1,p_1q_1) $ and $ \displaystyle \frac{Y_2}{n_2} = N(p_2,p_2q_2) $, created from population distributions which are Bernoulli distributions.

Note that $Y_1$ represents the sum of successes in a sample set, and thus $\dfrac{Y_1}{n_1}$ represents sample proportions. For example, for any kth sample set of $\dfrac{Y_1}{n_1}$, we calculate sample proportion statistic, $\dfrac{Y_{1k}}{n_1} = \dfrac {1}{n} \sum\limits_{i=1}^n Y_{1ki}$, where $Y_{1ki}$ is $i$th sample in $k$th sample set of sampling distribution described by $\dfrac{Y_1}{n_1}$. Similarly for $\dfrac{Y_2}{n_2}$

We could then calculate CI as below, $$ \begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\frac{Y_1}{n_1} - \frac{Y_2}{n_2}) - (p_1 - p_2) }{\sqrt{ {\frac{p_1q_1}{n_1}} + {\frac{p_2q_2}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) = 1-\alpha \nonumber \end{align} $$

Assuming $\sigma$ unknown

Most of the times in reality, the population paramters are not known. So when the sample sizes $n,m$ are sufficiently large, we could use sample statistics ($\frac{\hat{p_1}\hat{q_1}}{n1},\frac{\hat{p_2}\hat{q_2}}{n2}$) in place of ($\frac{p_1q_1}{n1},\frac{p_2q_2}{n2}$). This results in further approximation of our confidence intervals. Thus when a sample is observed, we have statistics

$\hat{p_1} = \dfrac{y_1}{n_1} , \hat{q_1} = 1 - \dfrac{y_1}{n_1}$,
$\hat{p_2} = \dfrac{y_2}{n_2} , \hat{q_2} = 1 - \dfrac{y_2}{n_2}$,

Thus we could rewrite further as,

$$ \begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\hat{p_1} - \hat{p_2}) - (p_1 - p_2) }{\sqrt{ {\frac{\hat{p_1}\hat{q_1}}{n_1}} + {\frac{\hat{p_2}{\hat{q_2}}}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) \approx 1-\alpha \end{align} $$

When $n,m$ are small

This is where I am left without any further info I could not find online. When $n_1 < 30, n_2 < 30$, what do we do? Will it be t-distribution again just like for single population distribution? If so, how do we calculate degrees of freedom there?

  • 1
    $\begingroup$ See this paper by Alan Agresti and Brian Caffo. $\endgroup$
    – guy
    Oct 2, 2018 at 15:56
  • $\begingroup$ The paper look complicated for me to understand fully with all the jargon, and I would like to skip to conclusion due to lack of time. After skimming it, I learn, the authors do not favour any specifically, but touch upon Bayesian interval developed by Carles at the end for elementary level, but I am unable to infer that to my situation,esp its not specifically said its for small values. Can you please translate that to my question? $\endgroup$ Oct 2, 2018 at 17:05
  • 1
    $\begingroup$ Agresti and Caffo recommend using the interval you give above, with the $Z$ reference distribution, after adding one success and one failure to each of the groups (two success and two failures in total). This leads to confidence intervals with close to the nominal coverage. They show that, without this correction, the interval you give above performs extremely poorly. $\endgroup$
    – guy
    Oct 2, 2018 at 18:41

1 Answer 1


As I read the Agresti-Caffo paper (recommended by @guy), it seems to me that they do have a specific recommendation at the end of Section 1:

For Sample 1: Let $\check n_1 = n_1 + 4,$ and $\check p_1 = (X_1 + 2)/\check n_1.$

For Sample 2: Let $\check n_2 = n_2 + 4,$ and $\check p_2 = (X_2 + 2)/\check n_2.$

To get a 95% CI for the difference $p_1 - p_2$ in population proportions, use their equation (2) with $\check n_1$ in place of $n_1,$ $\check n_2$ in place of $n_2,$ $\check p_1$ in place of $\hat p_1,$ $\check p_2$ in place of $\hat p_2,$ and $Z_{\alpha/2} = 1.96.$ That is,

$$\check p_1 - \check p_2 \pm 1.96\sqrt{\frac{\check p_1(1-\check p_1)}{\check n_1} + \frac{\check p_2(1-\check p_2)}{\check n_2}}.$$

Notes: You are correct that the form of the CI you write in your question runs into difficulties because (a) it relies on a normal approximation and (b) it relies on using estimates $\hat p_1$ and $\hat p_2$ for unknown $p_1$ and $p_2,$ respectively, to find the standard error. Without correction, these approximations do not work well with one another--especially when $p_1$ or $p_2$ is small. The Agresti-style CI may not have precisely the claimed "95%" coverage probability, but it is a great improvement on the Wald interval (mainly intended for use when $n$ is very large and $p_1$ and $p_2$ are both reasonably near 1/2).

Because (corrected) normal approximations are used in the style of interval Agresti and Caffo recommend, the t distribution is not involved, and you do not need to deal with 'degrees of freedom'.

  • $\begingroup$ great, thank you. I think this should do. the X1 and X2 are sample means? $\endgroup$ Oct 6, 2018 at 15:05
  • $\begingroup$ $X_1$ and $X_2$ are counts of successes. $\endgroup$
    – BruceET
    Oct 6, 2018 at 17:35
  • $\begingroup$ oh got it. one minor question. why are you inverting the cap above the letters? $\endgroup$ Oct 7, 2018 at 6:23
  • $\begingroup$ The most common estimate of Binomial success probability $p$ is $\hat p = X/n,$ where $X$ is the observed number of successes in $n$ trials. The Agresti-Coull style of confidence interval requires a slightly different estimate, so I use notation $\check p$ for that--according to the formula in my Answer. $\endgroup$
    – BruceET
    Oct 7, 2018 at 6:27
  • $\begingroup$ Can we also use Welch's t interval? $\endgroup$ Oct 9, 2018 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.