# Z-score computing for two sample proportion comparison

Assume I aggregated the following results:

| Treatment | Count | Total | Proportion |
|-----------|-------|-------|------------|
| Control   | $c_n$ | $c_t$ | $p_c$      |
| Variant   | $v_n$ | $v_t$ | $p_v$      |


It seems like I can compute the $z$-score in one of two methods:

Option 1 / Pooled (link): $z$-statistic is given by $$z = \frac{p_c - p_v}{\sqrt{\frac{\hat{p}(1-\hat{p})}{c_t} + \frac{\hat{p}(1-\hat{p})}{v_t} }}$$ where $p_c$ is the proportion of the control group and $p_v$ is the one of the variant group. Furthermore, $$\hat{p} = \frac{c_t p_c + v_t p_v}{c_t + v_t}$$ is the pooled proportion where $c_t$ and $v_t$ are the sample sizes of the control and variant groups, respectively.

Option 2 / Unpooled (link): Here the $z$-score is given by:

$$z = \frac{p_c - p_v}{\sqrt{SE_c^2 + SE_v^2}}$$

where

$$SE_c = \sqrt{\frac{p_c(1-p_c)}{c_t}} \quad ; \quad SE_v = \sqrt{\frac{p_v(1-p_v)}{v_t}}$$

My questions are:

• Under what circumstances should I use the one or the other? My understanding tells me it depends on my knowing/assumption on the variance. If I assume it is the same or not big of a difference, then I should use the pooled and the unpooled otherwise.
• When would the difference between the two be significant?
• Is there an implemented test in Python where I can choose which approach to use? In statsmodels.stats.proportion it seems like only the pooled option is available.

### Edit:

• Related questions:
• Also found an interesting video; however, it deals with the $t$-test. If I understand correctly I can port its message to the case I consider.

Ideally you want to calculate $$z = \frac{p_c-p_v}{\sqrt{\sigma_c^2+\sigma_v^2}} = \frac{p_c-p_v}{\sqrt{\frac{{p}(1-{p})}{c_t}+\frac{{p}(1-{p})}{v_t}}}$$ in which $$\sigma_c=p(1-p)/c_t$$ and $$\sigma_v=p(1-p)/v_t$$ relate to known variance.

But you do not know $$p$$, $$\sigma_c$$, $$\sigma_v$$ and instead estimate it either as

• pooled $$\hat p$$ $$z = \frac{p_c-p_v}{\sqrt{\frac{\hat{p}\left( 1-\hat{p} \right)}{c_t}+\frac{\hat{p}\left( 1-\hat{p} \right)}{v_t}}}$$
• or separately $$\hat{p}_c$$ and $$\hat{p}_v$$ $$z = \frac{p_c-p_v}{\sqrt{\frac{\hat{p}_c\left( 1-\hat{p}_c \right)}{c_t}+\frac{\hat{p}_v \left( 1-\hat{p}_v \right)}{v_t}}}$$

The effect of using estimates $$\hat\sigma_c$$ and $$\hat\sigma_v$$ is that the calculated z-score (which should actually be calculated with a known variance, and these z-scores are just approximations) is in effect not precisely/exactly distributed as a standard normal $$N(0,1)$$.

• One way that people deal with this, is to use an alternative distribution that incorporates this estimated deviation. You have found this yourself in the t-distribution expressing the distribution of (standardized) sample means. The t-distribution is a (complicated) variation of the z-distribution.

• Another way is to accept the difference. This may be valid when the number of measurements is high enough. Then the distribution, which is not really (exactly) a z-distribution, becomes more and more like a z-distribution.

The convergence of the (fake) z-score to resemble a z-distribution, goes faster when the $$p$$ is estimated more precisely. This is the case when you use the pooled proportion.

So the pooled variance is better when you want to make an expression that approximates a z-distribution as much as possible. However, if you allow yourself to jump trough a lot of different hoops, and come up with an (more complex) alternative to the z-test, then possibly the un-pooled statistic might give you a better power (I do not know this and would have to test it, but I imagine it could), that means it could provide more often a positive result when the proportions are different.

• What do you mean by "if you allow yourself to jump trough a lot of different hoops, and come up with an (more complex) alternative to the z-test"? Can you elaborate please? Is the choosing of pooled vs. non-pooled has to depend on the alternative hypothesis (one vs. two sided)? – Dror Atariah Feb 27 '18 at 12:23
• The z that you calculate is only approximately standard normal distributed. The pooled version is a better approximate than the un-pooled version, so therefore it is better. But possibly you might use something else than the z-score and the normal distribution in the design of your tests (which will be complicated and that is the reference to the hoops), in which case the un-pooled version might provide more power (more likely to reject $H_0$ when $H_0$ is false). – Sextus Empiricus Feb 27 '18 at 13:55
• Can you please add a pointer to your favorite reference discussing this point? – Dror Atariah Feb 28 '18 at 6:50
• This point about the alternative was purely hypothetical (so no references). But what I had in mind was en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma maybe that is a sufficient additional information for you. So the unpooled z-score is not correctly expressing the p-value and not preferable, but maybe when we correct the unpooled z-score or confidence intervals (or not correct at all since these levels are arbitrary anyway) they might give you a measure with better power. – Sextus Empiricus Feb 28 '18 at 8:45