# Two-sample difference in proportions: Why isn't estimating the standard error a problem for normality?

First, this seems like a very common question, but I do promise that I've searched for a long time through other similar posts and haven't quite found a solution that seems to provide an answer. (This and this come very close, but I don't think either quite answers it!).

I'll be very thorough in explaining my confusion, by comparing it first to the one-sample proportion case. We could test whether the true proportion $$p$$ equals some value $$c$$.

$$H_0: p = c.$$

$$H_A: p \neq c.$$

For $$np$$ and $$n(1-p)$$ sufficiently large, the binomial approximation to the normal gives us $$\hat{p} \sim N(p, p(1-p)/n)$$. Thus, when we compute the test statistic, $$Z := \frac{\hat{p} - c}{\sqrt{c(1-c)/n}}.$$ Under the null hypothesis this is distributed as standard normal. Crucially, we are not estimating the standard error–it is determined by the null hypothesis!

Now, we instead consider the two-sample case, where we want to run a hypothesis test on the difference in proportions.

$$H_0: p_1 - p_2 = 0.$$

$$H_A: p_1 - p_2 \neq 0.$$

The same binomial approximation gives us $$\hat{p}_i \sim N(p_i, p_i(1-p_i)/n_i)$$, $$i=1,2$$. Then, if $$\tilde{p}$$ is the pooled proportion ($$\hat{p} = (x_1 + x_2)/(n_1+n_2)$$), I know that our test statistic is given by $$Z := \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\tilde{p}(1-\tilde{p})(1/n_1 + 1/n_2)}}.$$

This is the crucial part I do not follow. In the one-sample case, we did not estimate the standard error–it was determined by the null. Hence, citing the standard normal makes sense. In the two-sample case, we do have to estimate the standard error, using our pooled proportion! So, basically, shouldn't we have to make some sort of adjustment to reflect this? (I.e., like what we do with a t-test).

So, how can I explain this? Why can we still cite the standard normal, even though we are using a data-based estimate for its standard error?

While the question is analogous to that of a t-test, I understand why we can't just use a t-test (we don't have the other assumptions met). My best attempt at an answer so far is simply:

“Yes, we are estimating the standard error, but that approximation is just swallowed up by our original normal approximation to the binomial, because that only works at high degrees of freedom anyways"?

Is there a better explanation that that? It feels intuitive that some adjustment would be necessary, but we don't make one.

For a brief explanation of why the two links I cited felt like they didn't quite cover this...

@glen_b's answer is very good, and clearly explains why the theoretical assumptions required for a t-test wouldn't hold here. However, I don't see it explain this exact issue. It mentions both one and two sample proportion tests, and my confusion is that they don't feel like they are the same on this front. But this helps inform my best guess above, which is that for sample sizes large enough for our normality approximation, estimating the standard error is basically irrelevant.

Similarly, @whuber's very clearly shows (with plots) how the student-t distribution doesn't approximate our test statistic any better (for moderate sample size) than the simple standard normal.

So, I understand why the student-t distribution isn't a better choice here. But my lingering confusion is simply: even if the student-t isn't the right fix, what's the best way to describe why we don't provide any sort of adjustment here? I see the one sample and two sample case described equivalently–"it's normal because the variance is determined by the mean". But the cases seem quite different–in one, we estimate the standard error, in the other, we don't. Is the answer simply "once we are already approximating the binomial proportion difference with a normal, the estimation of the standard error is trivial relative to that approximation, so we can ignore it"?

• It's hard to tell what you conceive the issue to be, but maybe it's addressed at stats.stackexchange.com/questions/411699/…? Although you linked to it and claim it doesn't answer your question, please tell us what remains to be said. – whuber Aug 15 at 16:24
• Ah sorry forgot to add an explanation for why I felt that link didn't answer what I was getting at. I edited the end of the post to clarify. In short, both your and glen_b's great answers explain why the student t doesn't work (his from the assumptions, yours with plots). But I'm trying to understand how we should explain why we don't make any adjustment, despite estimating the standard error. As I continue to read these answers, I think it's just "the error from estimating the intermediate parameter is trivial relative to the normality approximation", but I wondered if there was more. – Ziddletwix Aug 15 at 18:02

As in the answer you link to, you again use Slutsky's theorem, specifically the third (ratio) form at the Wikipedia link.

If you write $$Z_n$$ as $$X_n/Y_n$$ where

$$X_n = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{p_0(1-p_0)(1/n_1 + 1/n_2)}}$$

$$Y_n = \frac{\sqrt{\tilde{p}(1-\tilde{p})}}{\sqrt{p_0(1-p_0)}}$$

where $$p_0$$ is the common population proportion under the null and $$n_1$$ and $$n_2$$ increase proportionally* (or, alternatively by letting $$n$$ be the smaller of $$n_1$$ and $$n_2$$ instead) then the theorem should apply, and $$Y_n$$ converges to $$1$$, so as $$n\to\infty$$ the sequence $$Z_n$$ converges to the same distribution $$X_n$$ does (i.e. to the standard normal distribution).

* this part could be formalized, in a number of similar possible ways, relating a sequence of $$n_1$$ and $$n_2$$ values to $$n$$ while holding them in proportion

• Ok thanks, this probably covers it. This lays out the theoretical argument for the asymptotic convergence of the test statistic. I guess part of my confusion was a bit more colloquial, basically "Why don't we feel obligated to adjust the test statistic, now that we have to estimate the standard error as well", because after all, the t-test is also asymptotically equivalent to the z-test! But, if the answer is just this asymptotic convergence, then I assume the answer to my informal version is just "estimating the standard error is trivial relative to the main approximation binomial/normal". – Ziddletwix Aug 15 at 18:38
• As in whubers answer to the question you link to, it would be be similar in the two sample case that the difference between the t and the z will tend to be smaller than the inaccuracy in the approximation of the discrete difference in proportions, though I haven't done much investigation. – Glen_b Aug 15 at 23:38