What formula for a Confidence Interval of the difference in proportions when sample sizes are small

Question

Suppose that we are interested in comparing two approximately normal sampling distributions described by random variables $ \displaystyle \frac{Y_1}{n_1} = N(p_1,p_1q_1) $ and $ \displaystyle \frac{Y_2}{n_2} = N(p_2,p_2q_2) $, created from population distributions which are Bernoulli distributions.

Note that $Y_1$ represents the sum of successes in a sample set, and thus $\dfrac{Y_1}{n_1}$ represents sample proportions. For example, for any kth sample set of $\dfrac{Y_1}{n_1}$, we calculate sample proportion statistic, $\dfrac{Y_{1k}}{n_1} = \dfrac {1}{n} \sum\limits_{i=1}^n Y_{1ki}$, where $Y_{1ki}$ is $i$th sample in $k$th sample set of sampling distribution described by $\dfrac{Y_1}{n_1}$. Similarly for $\dfrac{Y_2}{n_2}$

We could then calculate CI as below, $$ \begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\frac{Y_1}{n_1} - \frac{Y_2}{n_2}) - (p_1 - p_2) }{\sqrt{ {\frac{p_1q_1}{n_1}} + {\frac{p_2q_2}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) = 1-\alpha \nonumber \end{align} $$

Assuming $\sigma$ unknown

Most of the times in reality, the population paramters are not known. So when the sample sizes $n,m$ are sufficiently large, we could use sample statistics ($\frac{\hat{p_1}\hat{q_1}}{n1},\frac{\hat{p_2}\hat{q_2}}{n2}$) in place of ($\frac{p_1q_1}{n1},\frac{p_2q_2}{n2}$). This results in further approximation of our confidence intervals. Thus when a sample is observed, we have statistics

$\hat{p_1} = \dfrac{y_1}{n_1} , \hat{q_1} = 1 - \dfrac{y_1}{n_1}$,
$\hat{p_2} = \dfrac{y_2}{n_2} , \hat{q_2} = 1 - \dfrac{y_2}{n_2}$,

Thus we could rewrite further as,

$$ \begin{align} Pr\Bigg( -z_{\frac{\alpha}{2}} \leq \dfrac{(\hat{p_1} - \hat{p_2}) - (p_1 - p_2) }{\sqrt{ {\frac{\hat{p_1}\hat{q_1}}{n_1}} + {\frac{\hat{p_2}{\hat{q_2}}}{n_2}} }} \leq z_{\frac{\alpha}{2}}\Bigg) \approx 1-\alpha \end{align} $$

When $n,m$ are small

This is where I am left without any further info I could not find online. When $n_1 < 30, n_2 < 30$, what do we do? Will it be t-distribution again just like for single population distribution? If so, how do we calculate degrees of freedom there?

Answer 1

As I read the Agresti-Caffo paper (recommended by @guy), it seems to me that they do have a specific recommendation at the end of Section 1:

For Sample 1: Let $\check n_1 = n_1 + 4,$ and $\check p_1 = (X_1 + 2)/\check n_1.$

For Sample 2: Let $\check n_2 = n_2 + 4,$ and $\check p_2 = (X_2 + 2)/\check n_2.$

To get a 95% CI for the difference $p_1 - p_2$ in population proportions, use their equation (2) with $\check n_1$ in place of $n_1,$ $\check n_2$ in place of $n_2,$ $\check p_1$ in place of $\hat p_1,$ $\check p_2$ in place of $\hat p_2,$ and $Z_{\alpha/2} = 1.96.$ That is,

$$\check p_1 - \check p_2 \pm 1.96\sqrt{\frac{\check p_1(1-\check p_1)}{\check n_1} + \frac{\check p_2(1-\check p_2)}{\check n_2}}.$$

Notes: You are correct that the form of the CI you write in your question runs into difficulties because (a) it relies on a normal approximation and (b) it relies on using estimates $\hat p_1$ and $\hat p_2$ for unknown $p_1$ and $p_2,$ respectively, to find the standard error. Without correction, these approximations do not work well with one another--especially when $p_1$ or $p_2$ is small. The Agresti-style CI may not have precisely the claimed "95%" coverage probability, but it is a great improvement on the Wald interval (mainly intended for use when $n$ is very large and $p_1$ and $p_2$ are both reasonably near 1/2).

Because (corrected) normal approximations are used in the style of interval Agresti and Caffo recommend, the t distribution is not involved, and you do not need to deal with 'degrees of freedom'.