# What formula for a Confidence Interval of the difference in proportions when sample sizes are small

Suppose that we are interested in comparing two approximately normal sampling distributions described by random variables $$\displaystyle \frac{Y_1}{n_1} = N(p_1,p_1q_1)$$ and $$\displaystyle \frac{Y_2}{n_2} = N(p_2,p_2q_2)$$, created from population distributions which are Bernoulli distributions.

Note that $$Y_1$$ represents the sum of successes in a sample set, and thus $$\dfrac{Y_1}{n_1}$$ represents sample proportions. For example, for any kth sample set of $$\dfrac{Y_1}{n_1}$$, we calculate sample proportion statistic, $$\dfrac{Y_{1k}}{n_1} = \dfrac {1}{n} \sum\limits_{i=1}^n Y_{1ki}$$, where $$Y_{1ki}$$ is $$i$$th sample in $$k$$th sample set of sampling distribution described by $$\dfrac{Y_1}{n_1}$$. Similarly for $$\dfrac{Y_2}{n_2}$$

We could then calculate CI as below, \begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\frac{Y_1}{n_1} - \frac{Y_2}{n_2}) - (p_1 - p_2) }{\sqrt{ {\frac{p_1q_1}{n_1}} + {\frac{p_2q_2}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) = 1-\alpha \nonumber \end{align}

Assuming $$\sigma$$ unknown

Most of the times in reality, the population paramters are not known. So when the sample sizes $$n,m$$ are sufficiently large, we could use sample statistics ($$\frac{\hat{p_1}\hat{q_1}}{n1},\frac{\hat{p_2}\hat{q_2}}{n2}$$) in place of ($$\frac{p_1q_1}{n1},\frac{p_2q_2}{n2}$$). This results in further approximation of our confidence intervals. Thus when a sample is observed, we have statistics

$$\hat{p_1} = \dfrac{y_1}{n_1} , \hat{q_1} = 1 - \dfrac{y_1}{n_1}$$,
$$\hat{p_2} = \dfrac{y_2}{n_2} , \hat{q_2} = 1 - \dfrac{y_2}{n_2}$$,

Thus we could rewrite further as,

\begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\hat{p_1} - \hat{p_2}) - (p_1 - p_2) }{\sqrt{ {\frac{\hat{p_1}\hat{q_1}}{n_1}} + {\frac{\hat{p_2}{\hat{q_2}}}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) \approx 1-\alpha \end{align}

When $$n,m$$ are small

This is where I am left without any further info I could not find online. When $$n_1 < 30, n_2 < 30$$, what do we do? Will it be t-distribution again just like for single population distribution? If so, how do we calculate degrees of freedom there?

• See this paper by Alan Agresti and Brian Caffo. – guy Oct 2 '18 at 15:56
• The paper look complicated for me to understand fully with all the jargon, and I would like to skip to conclusion due to lack of time. After skimming it, I learn, the authors do not favour any specifically, but touch upon Bayesian interval developed by Carles at the end for elementary level, but I am unable to infer that to my situation,esp its not specifically said its for small values. Can you please translate that to my question? – Parthiban Rajendran Oct 2 '18 at 17:05
• Agresti and Caffo recommend using the interval you give above, with the $Z$ reference distribution, after adding one success and one failure to each of the groups (two success and two failures in total). This leads to confidence intervals with close to the nominal coverage. They show that, without this correction, the interval you give above performs extremely poorly. – guy Oct 2 '18 at 18:41

As I read the Agresti-Caffo paper (recommended by @guy), it seems to me that they do have a specific recommendation at the end of Section 1:

For Sample 1: Let $$\check n_1 = n_1 + 4,$$ and $$\check p_1 = (X_1 + 2)/\check n_1.$$

For Sample 2: Let $$\check n_2 = n_2 + 4,$$ and $$\check p_2 = (X_2 + 2)/\check n_2.$$

To get a 95% CI for the difference $$p_1 - p_2$$ in population proportions, use their equation (2) with $$\check n_1$$ in place of $$n_1,$$ $$\check n_2$$ in place of $$n_2,$$ $$\check p_1$$ in place of $$\hat p_1,$$ $$\check p_2$$ in place of $$\hat p_2,$$ and $$Z_{\alpha/2} = 1.96.$$ That is,

$$\check p_1 - \check p_2 \pm 1.96\sqrt{\frac{\check p_1(1-\check p_1)}{\check n_1} + \frac{\check p_2(1-\check p_2)}{\check n_2}}.$$

Notes: You are correct that the form of the CI you write in your question runs into difficulties because (a) it relies on a normal approximation and (b) it relies on using estimates $$\hat p_1$$ and $$\hat p_2$$ for unknown $$p_1$$ and $$p_2,$$ respectively, to find the standard error. Without correction, these approximations do not work well with one another--especially when $$p_1$$ or $$p_2$$ is small. The Agresti-style CI may not have precisely the claimed "95%" coverage probability, but it is a great improvement on the Wald interval (mainly intended for use when $$n$$ is very large and $$p_1$$ and $$p_2$$ are both reasonably near 1/2).

Because (corrected) normal approximations are used in the style of interval Agresti and Caffo recommend, the t distribution is not involved, and you do not need to deal with 'degrees of freedom'.

• great, thank you. I think this should do. the X1 and X2 are sample means? – Parthiban Rajendran Oct 6 '18 at 15:05
• $X_1$ and $X_2$ are counts of successes. – BruceET Oct 6 '18 at 17:35
• oh got it. one minor question. why are you inverting the cap above the letters? – Parthiban Rajendran Oct 7 '18 at 6:23
• The most common estimate of Binomial success probability $p$ is $\hat p = X/n,$ where $X$ is the observed number of successes in $n$ trials. The Agresti-Coull style of confidence interval requires a slightly different estimate, so I use notation $\check p$ for that--according to the formula in my Answer. – BruceET Oct 7 '18 at 6:27
• Can we also use Welch's t interval? – Parthiban Rajendran Oct 9 '18 at 13:26