10
$\begingroup$

The z-test to compare two proportions is $\newcommand{\p}{\hat{p}}\newcommand{\v}{\mathrm{Var}} z=\frac{\p_1-\p_2}{\sqrt{\v(\p_1-\p_2)}}$. Usually it is defined that

$$\v(\p_1-\p_2)=\p(1-\hat{p})(1/n_1+1/n_2),$$

where

$$\p=\frac{n_1 \p_1+n_2 \p_2}{n_1+n_2}.$$

Is there any written reference that legitimizes me instead to use the unpooled variance, that is

$$\v(\p_1-\p_2)=\frac{\p_1(1-\p_1)}{n_1}+\frac{\p_2(1-\p_2)}{n_2}?$$

Improve this question
$\endgroup$
8
$\begingroup$

There is quite a bit of discussion about this on the AP site.

You can use whatever statistic you want, provided that you are clear about what you do and look at the appropriate null distribution to calculate p-values or thresholds.

But some statistics are better than others; in this case you'd be looking for (a) null distribution easily calculated and (b) power to detect difference.

But I don't know why you'd favor the unpooled variance over the pooled variance for the test, though it could be preferred in calculating a confidence interval for the difference.

Improve this answer
$\endgroup$
  • $\begingroup$ +1 That's a good discussion you found. However, it seems to fall short of really addressing the question, which is whether somehow the pooled statistic could be corrected to give the desired test size and--perhaps--yield greater power. To resolve this issue, I have provided a separate reply. $\endgroup$ – whuber♦ Oct 18 '11 at 16:05
  • 1
    $\begingroup$ Your link doesn't go to a discussion; it goes to a page with Charles Peltier's viewpoint. Not sure why this is the selected answer as it doesn't answer anything for me. Use whatever statistic isn't concrete enough. $\endgroup$ – Jarad Jul 27 '16 at 15:40
  • 2
    $\begingroup$ @Jarad One definition of the word "discussion" is "a detailed treatment of a particular topic"; that's what I meant. The selected answer is chosen by the person asking the question. By "use whatever statistic you want", I was referring to the "...reference that legitimizes me..." part of the question. $\endgroup$ – Karl Jul 28 '16 at 16:31
10
$\begingroup$

The unpooled variance tends to be too small. This is because under the null hypothesis there will still be chance variation in the two observed proportions, although the underlying probabilities are equal. This chance variation contributes to the pooled variance but not to the unpooled variance.

As a result, $z$ for the unpooled statistic does not even approximately have a standard normal distribution. For instance, when $n_1 = n_2$ and the true probabilities are both $1/2$, the variance of $z$ is only $1/2$ instead of $1$. By using tables of the standard normal distribution, you will get incorrect p-values: they will tend to be artificially small, too often rejecting the null when the evidence is not really there.

Nevertheless, one wonders whether this could be corrected. It can. The question becomes whether a corrected value of $z$, based on unpooled estimates, could have greater power to detect deviations from the null hypothesis. A few quick simulations suggest this is not the case: the pooled test (compared to a properly adjusted unpooled test) has a better chance of rejecting the null whenever the null is false. Therefore I haven't bothered to work out the formula for the unpooled correction; it seems pointless.

In summary, the unpooled test is wrong, but with an appropriate correction, it can be made legitimate. However, it appears to be inferior to the pooled test.

Improve this answer
$\endgroup$
  • $\begingroup$ You say "For instance, when $n_1=n_2$ and the true probabilities are both 1/2, the variance of z is only 1/2 instead of 1." But if the unpooled variance is too small, the variance of z should be too large, and I would think it would be only slightly too large. $\endgroup$ – Karl Oct 18 '11 at 16:35
  • $\begingroup$ Forgive me but I am unable to follow your example. Why should the variance of $z$ be 1? Which values are you assuming for $\hat{p}_1$ and $\hat{p}_2$? $\endgroup$ – glassy Oct 20 '11 at 11:46
  • $\begingroup$ @glassy $z$ has (asymptotically) unit variance by construction: the difference $\hat{p_1}-\hat{p_1}$ has been standardized by dividing it by its estimated variance. $\endgroup$ – whuber♦ Oct 20 '11 at 15:23
  • $\begingroup$ I don't want to bother you but really I do not undertand why if $z$ has unit variance by construction you state that its variance can be $1/2$. It seems to me that its variance is equal to $\hat{p}(1-\hat{p})\frac{2}{n}$ in a case and $\frac{\hat{p}_1(1-\hat{p}_1)}{n}+\frac{\hat{p}_2(1-\hat{p}_2)}{n}$ in the other. Sorry, I do not understand how these quantities have a 2:1 ratio. Indeed, in the case $\hat{p}_1=\hat{p}_2$ they are the same. $\endgroup$ – glassy Oct 20 '11 at 20:23
  • 1
    $\begingroup$ @Amonet All the statements I made in this post were derived from the simulations and reasoning I have described. I did not refer to any authorities to deduce them. $\endgroup$ – whuber♦ Nov 3 '20 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.