|
The z-test to compare two proportions is $\newcommand{\p}{\hat{p}}\newcommand{\v}{\mathrm{Var}} z=\frac{\p_1-\p_2}{\sqrt{\v(\p_1-\p_2)}}$. Usually it is defined that $$\v(\p_1-\p_2)=\p(1-\hat{p})(1/n_1+1/n_2),$$ where $$\p=\frac{n_1 \p_1+n_2 \p_2}{n_1+n_2}.$$ Is there any written reference that legitimizes me instead to use the unpooled variance, that is $$\v(\p_1-\p_2)=\frac{\p_1(1-\p_1)}{n_1}+\frac{\p_2(1-\p_2)}{n_2}?$$ |
||||
|
|
|
There is quite a bit of discussion about this on the AP site. You can use whatever statistic you want, provided that you are clear about what you do and look at the appropriate null distribution to calculate p-values or thresholds. But some statistics are better than others; in this case you'd be looking for (a) null distribution easily calculated and (b) power to detect difference. But I don't know why you'd favor the unpooled variance over the pooled variance for the test, though it could be preferred in calculating a confidence interval for the difference. |
|||
|
|
The unpooled variance tends to be too small. This is because under the null hypothesis there will still be chance variation in the two observed proportions, although the underlying probabilities are equal. This chance variation contributes to the pooled variance but not to the unpooled variance. As a result, $z$ for the unpooled statistic does not even approximately have a standard normal distribution. For instance, when $n_1 = n_2$ and the true probabilities are both $1/2$, the variance of $z$ is only $1/2$ instead of $1$. By using tables of the standard normal distribution, you will get incorrect p-values: they will tend to be artificially small, too often rejecting the null when the evidence is not really there. Nevertheless, one wonders whether this could be corrected. It can. The question becomes whether a corrected value of $z$, based on unpooled estimates, could have greater power to detect deviations from the null hypothesis. A few quick simulations suggest this is not the case: the pooled test (compared to a properly adjusted unpooled test) has a better chance of rejecting the null whenever the null is false. Therefore I haven't bothered to work out the formula for the unpooled correction; it seems pointless. In summary, the unpooled test is wrong, but with an appropriate correction, it can be made legitimate. However, it appears to be inferior to the pooled test. |
|||||||||||
|
