Assume I aggregated the following results:

| Treatment | Count | Total | Proportion |
| Control   | $c_n$ | $c_t$ | $p_c$      |
| Variant   | $v_n$ | $v_t$ | $p_v$      |

It seems like I can compute the $z$-score in one of two methods:

Option 1 / Pooled (link): $z$-statistic is given by $$z = \frac{p_c - p_v}{\sqrt{\frac{\hat{p}(1-\hat{p})}{c_t} + \frac{\hat{p}(1-\hat{p})}{v_t} }} $$ where $p_c$ is the proportion of the control group and $p_v$ is the one of the variant group. Furthermore, $$\hat{p} = \frac{c_t p_c + v_t p_v}{c_t + v_t}$$ is the pooled proportion where $c_t$ and $v_t$ are the sample sizes of the control and variant groups, respectively.

Option 2 / Unpooled (link): Here the $z$-score is given by:

$$z = \frac{p_c - p_v}{\sqrt{SE_c^2 + SE_v^2}}$$


$$ SE_c = \sqrt{\frac{p_c(1-p_c)}{c_t}} \quad ; \quad SE_v = \sqrt{\frac{p_v(1-p_v)}{v_t}} $$

My questions are:

  • Under what circumstances should I use the one or the other? My understanding tells me it depends on my knowing/assumption on the variance. If I assume it is the same or not big of a difference, then I should use the pooled and the unpooled otherwise.
  • When would the difference between the two be significant?
  • Is there an implemented test in Python where I can choose which approach to use? In statsmodels.stats.proportion it seems like only the pooled option is available.



Ideally you want to calculate $$z = \frac{p_c-p_v}{\sqrt{\sigma_c^2+\sigma_v^2}} = \frac{p_c-p_v}{\sqrt{\frac{{p}(1-{p})}{c_t}+\frac{{p}(1-{p})}{v_t}}} $$ in which $\sigma_c=p(1-p)/c_t$ and $\sigma_v=p(1-p)/v_t$ relate to known variance.

But you do not know $p$, $\sigma_c$, $\sigma_v$ and instead estimate it either as

  • pooled $\hat p$ $$z = \frac{p_c-p_v}{\sqrt{\frac{\hat{p}\left( 1-\hat{p} \right)}{c_t}+\frac{\hat{p}\left( 1-\hat{p} \right)}{v_t}}} $$
  • or separately $\hat{p}_c$ and $\hat{p}_v$ $$z = \frac{p_c-p_v}{\sqrt{\frac{\hat{p}_c\left( 1-\hat{p}_c \right)}{c_t}+\frac{\hat{p}_v \left( 1-\hat{p}_v \right)}{v_t}}} $$

The effect of using estimates $\hat\sigma_c$ and $\hat\sigma_v$ is that the calculated z-score (which should actually be calculated with a known variance, and these z-scores are just approximations) is in effect not precisely/exactly distributed as a standard normal $N(0,1)$.

  • One way that people deal with this, is to use an alternative distribution that incorporates this estimated deviation. You have found this yourself in the t-distribution expressing the distribution of (standardized) sample means. The t-distribution is a (complicated) variation of the z-distribution.

  • Another way is to accept the difference. This may be valid when the number of measurements is high enough. Then the distribution, which is not really (exactly) a z-distribution, becomes more and more like a z-distribution.

    The convergence of the (fake) z-score to resemble a z-distribution, goes faster when the $p$ is estimated more precisely. This is the case when you use the pooled proportion.

So the pooled variance is better when you want to make an expression that approximates a z-distribution as much as possible. However, if you allow yourself to jump trough a lot of different hoops, and come up with an (more complex) alternative to the z-test, then possibly the un-pooled statistic might give you a better power (I do not know this and would have to test it, but I imagine it could), that means it could provide more often a positive result when the proportions are different.

| cite | improve this answer | |
  • $\begingroup$ What do you mean by "if you allow yourself to jump trough a lot of different hoops, and come up with an (more complex) alternative to the z-test"? Can you elaborate please? Is the choosing of pooled vs. non-pooled has to depend on the alternative hypothesis (one vs. two sided)? $\endgroup$ – Dror Atariah Feb 27 '18 at 12:23
  • $\begingroup$ The z that you calculate is only approximately standard normal distributed. The pooled version is a better approximate than the un-pooled version, so therefore it is better. But possibly you might use something else than the z-score and the normal distribution in the design of your tests (which will be complicated and that is the reference to the hoops), in which case the un-pooled version might provide more power (more likely to reject $H_0$ when $H_0$ is false). $\endgroup$ – Sextus Empiricus Feb 27 '18 at 13:55
  • $\begingroup$ Can you please add a pointer to your favorite reference discussing this point? $\endgroup$ – Dror Atariah Feb 28 '18 at 6:50
  • $\begingroup$ This point about the alternative was purely hypothetical (so no references). But what I had in mind was en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma maybe that is a sufficient additional information for you. So the unpooled z-score is not correctly expressing the p-value and not preferable, but maybe when we correct the unpooled z-score or confidence intervals (or not correct at all since these levels are arbitrary anyway) they might give you a measure with better power. $\endgroup$ – Sextus Empiricus Feb 28 '18 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.