Why is it for the pooled test of two proportions, it can only be used for a hypothesis test and not a confidence interval?

Question

I read here that when using the pooled test of 2 proportions, it can only yield the hypothesis test, and not the confidence interval. Why is that? It also says to use pooled when $H_0: p_1-p_2=0$, but does that mean when $H_0: p_1-p_2=a$ where $a\neq 0$ we will use pooled? If so why?

Answer 1

A basic question to ask is, how would you estimate the standard error of the difference in proportions?

The model that we have comprises two independent groups, each of which may be represented by a binomial random variable, say $X_1 \sim \operatorname{Binomial}(n_1, p_1)$, and $X_2 \sim \operatorname{Binomial}(n_2, p_2)$, and the natural point estimate for the difference $p_1 - p_2$ is $$\hat p_1 - \hat p_2 = \frac{X_1}{n_1} - \frac{X_2}{n_2}.$$ Here $X_1$, $X_2$ represent the within-group frequencies.

What is the variance of this estimator? It is $$\operatorname{Var}[\hat p_1 - \hat p_2] = \frac{p_1 (1-p_1)}{n_1} + \frac{p_2 (1-p_2)}{n_2},$$

under the assumption that $X_1$ and $X_2$ are independent. So it is also natural to use the estimate $$\widehat{\operatorname{SE}}[\hat p_1 - \hat p_2] = \sqrt{\frac{\hat p_1 (1- \hat p_1)}{n_1} + \frac{\hat p_2 (1 - \hat p_2)}{n_2}}$$ and the $100(1-\alpha)\%$ Wald confidence interval for the difference in proportions is $$\hat p_1 - \hat p_2 \pm z_{\alpha/2}^* \widehat{\operatorname{SE}}[\hat p_1 - \hat p_2],$$ which assumes that the statistic $\hat p_1 - \hat p_2$ is approximately normally distributed.

A good exercise is to perform simulations for various choices of $n_1, p_1, n_2, p_2$ and calculate the resulting confidence interval; then calculate the coverage probability of the estimate by counting the proportion of intervals that contain the difference $p_1 - p_2$. You will find that the actual coverage is consistent with the nominal coverage as long as the group sizes are sufficiently large and the parameters not too extreme. But you will also find that if you use a pooled estimate of the standard error, your coverage probability will be inadequate except when $p_1 \approx p_2$.

And this is the crux of the issue: you could construct an interval estimate for $p_1 - p_2$ that employs a pooled standard error, but it would only have the claimed coverage probability when the difference in proportions is zero. It will become increasingly poor as the true difference in proportions deviates from zero. And for this reason, it isn't a useful interval estimate, because the original intent is to make an inference about $p_1 - p_2$, and if you assume it is zero to begin with, there is no point in using data to estimate it.

Answer 2

The pooled estimate of the standard error is only valid if the two proportions have the same value (which is the assumption under the null hypothesis). So it is fine to use for a significance test of no difference. There would be no point calculating a confidence interval for the difference in proportions by a method (pooling) that would only be valid if they have the same value. Such a procedure fails the definition of a confidence interval.