Why a 95%CI for difference of proportions and a 2x2 Chi-square test of independence don't agree

Question

A 95% confidence interval for a difference of proportions and a 2x2 Chi-square test of independence both try to answer the same question, in an elementary situation, using the same distribution (namely the standard normal one).

When carrying out the computations, you would expect either that they're fundamentally different for whatever reason, or that they are completely equivalent even though presented differently. Yet it turns out that the mathematical expressions that one gets are almost identical, but not quite. So my question is:

Is there a deep reason for that?

Below are the actual computations. To make the computations simpler I will assume that two samples both of the same size $N$ were taken from two populations (say males and females), where $a$ males and $b$ females were left-handed.

$$\matrix{&\text{left}&\text{right}\\ \text{males}&a&N-a \\\text{females}&b & N-b}$$

On the one hand, the 95% C.I. is given by $$\frac{a-b}{N}\pm 1.96 \sqrt {\frac{a(N-a)}{N^3}+\frac{b(N-b)}{N^3}}$$

so that the null-hypothesis (the proportion of left-handed people is the same in both genders) is rejected when $$0\notin \text{C.I.} \Leftrightarrow \frac{(a-b)^2}{\frac{a(N-a)}{N}+\frac{b(N-b)}{N}}> 3.842$$ which you can simplify as $$0\notin \text{C.I.} \Leftrightarrow \frac{N(a-b)^2 }{(a+b)N-{\color{red}{(a^2+b^2)}}}> 3.842$$

In the Chi-square approach, only the red part will be slightly different, which I find mesmerizing. Usually it means that I've made a mistake somewhere, but it doesn't look like this is the case here.

On the other hand, the expected counts are $$\matrix{&\text{left}&\text{right}\\ \text{males}&\frac{a+b}{2}&N-\frac{a+b}{2} \\\text{females}&\frac{a+b}{2} & N-\frac{a+b}{2}}$$

and the Chi-square statistic is given by $$\chi^2 = 2\left[\frac{\left(\frac{a-b}{2}\right)^2}{\frac{a+b}{2}}+\frac{\left(\frac{a-b}{2}\right)^2}{N-\frac{a+b}{2}}\right]$$

so that the same null-hypothesis is rejected when

$$(a-b)^2 \times\left[\frac{1}{a+b}+\frac{1}{2N-a-b}\right] > 3.842$$ or $$ \left[\frac{N(a-b)^2}{(a+b)N-{\color{red}{\frac{(a+b)^2}{2}}}}\right] > 3.842$$

I would be happy if this was just some computational mistake, and I would be happy if there is an insightful explanation for this.

Answer 1

The OP is correct in expecting that a z-test of 2 proportions should give the exact same value as a $\chi^2$ test (again, comparing 2 proportions). This is because a $\chi^2$ distribution with $d.f.=1$ is exactly the square of a Z distribution. So the Z statistic should be the square root of the $\chi^2$ statistic, and the p-values should be the same...
So what is going on?

The issue is that there are many "flavors" of the z-test, the 2 major ones being using the pooled variance (since we assume, under the Null, that the 2 proportions are the same), or the unpooled variance (which does not assume equal proportions, and tends to give more consistent CI's (consistent with the p-value)). Then you also find so-called "(n-1) z tests", with or w/o pooled variances (using Bessel's correction), as well as z-tests with Yates continuity correction, also with or w/o pooled variance (Yates continuity correction gives p-values which are much closer to the Fisher-exact test) (and there may be others?). See e.g. here for an overview of the pooled vs. unpooled "flavors".

But the $\chi^2$ has only 1 "flavor", and the math works out as being the pooled version of the z test. But the OP used the unpooled flavor (which seems to be recommended for computing CI's: see e.g. the very bottom of this page). The pooled variance, using the OP's notation and scenario ends up being ${s_{pool}}^2=\dfrac {a+b} {2N}.(1-\dfrac {a+b} {2N})$, with $\dfrac {a+b} {2N}$ being the pooled proportion.

Now there seems to be some debate about whether one should use the pooled, or unpooled flavor. See e.g. here or here on CV (and these links have themselves links to further discussions). Most textbooks only present the pooled version (maybe because it exactly matches the $\chi^2$ test, as the OP expected?). Note that as $a$ and $b$ get both large, the difference between pooled/unpooled gets smaller (asymptotically, the pooled and unpooled variances are equal).

Answer 2

I don't think there's anything profound here. They are similar because they are tests of the same hypothesis and because there are relatively few things to "play with". They are different because they are different tests of the same hypothesis.

Then you could also look at exact versions, or the Yates' correction, and so on and get further, slightly different formulas. And there are other measures as well.