2
$\begingroup$

The author of a famous online course takes a problem where 642 of 1000 polled men and 591 of 1000 polled women responded yes. The question is if men are more likely to agree or not, using 95% confidence interval. He starts by computing the difference of the means, $\bar p_1 - \bar p_2 = .642 -. 591 = 0.051$ and its standard deviation (i.e. the standard error of the .051), which is a sum of variances $\sigma = \sigma(\bar p_1) + \sigma(\bar p_2) = \sqrt{1/1000 (p_1(1-p_1) + p_2(1-p_2))} = 0.021715$. This difference of means is normally distributed and, according to z-table, 1.96 standard deviations contain 95% of such mean differences. We thus can say that the difference is 0.051 ± 1.96 * 0.021715 = 0.051 ± .042562 or there is 95% confidence that $p_1-p_2$ lies between .008 and .094. Since zero is excluded from the region, the 95% confidence interval is purely positive and we are sure that men are more likely to say yes than women.

The problem is that after this conclusion, author conducts something similar yet different: he tests the hypothesis that $\bar p_1 = \bar p_2$! That is, he takes the population average, $p = (\bar p_1 + \bar p_2)/2$, computes its error by summing variances for men and women. BTW, I wonder why claiming that there are 2000 individuals averaged he uses formula $\sqrt{2p(1-p) \over 1000} = 0.0217$ instead of $\sqrt{p(1-p) \over 2000}$? The difference is 2x. Anyway, he takes the z-score = $0.051/0.0217=2.35$ deviations afterwards, which is outside the 95% confidence interval (of 1.96 deviations) and, therefore, it is unlikely that means, $\bar p_1$ and $\bar p_2$, are equal.

I do not understand: what is the difference between the methods and which of the tests should I use?

$\endgroup$
  • $\begingroup$ Both methods are concluding that the two means are significantly different. So, what is the issue? $\endgroup$ – Peter Flom Nov 25 '13 at 15:27
  • $\begingroup$ @PeterFlom Do you mean that they cannot disagree? That could be the answer! $\endgroup$ – Val Nov 25 '13 at 15:43
  • $\begingroup$ Do you mean: How do I best test the difference between two proportions? $\endgroup$ – Peter Flom Nov 25 '13 at 15:56
  • $\begingroup$ Yes, I do not understand why the hypothesis testing was used after you have already tested the difference and if methods are not essentially identical then they may give contradictory answers. Isn't this a problem? $\endgroup$ – Val Nov 25 '13 at 16:00
3
$\begingroup$

You don't do tests of equality of sample proportions at all. Indeed, you can tell at a glance whether sample proportions differ!

It's population proportions you test the equality of. It's the fact that you usually can't observe population proportions that makes statistical inference necessary for this.

Under $H_0$, where the population proportions are equal, there are several possible estimators of the standard error of the difference in sample proportion.

They make different use of the available information, and may not be equally efficient under small deviations from the assumptions and so might have different power characteristics in small samples

Let $p_1$ and $p_2$ be the population proportions (I'd normally follow convention and use Greek symbols for population quantities but I am trying to follow your notation as closely as I reasonably can).

The null is that $p_1 = p_2 = p$. The alternative is that $p_1$ and $p_2$ differ.

The way hypothesis testing works is you look at the distribution of a test statistic under the null hypothesis, and by comparing the test statistic computed on the sample at hand, you can compute the probability (under the null) of a result at least as extreme* as the one you got -- the p-value.

*(that is, at least different from what you'd expect, under the null)

The standard error of $\bar p_i$ is $p_i (1-p_i)/n_i$, but under the null, $p_1 = p_2 = p$, so we are left with either estimating the standard error of $\bar{p}_1 - \bar{p}_2$ as (i) $\bar{p}_1 (1-\bar{p}_1)/n_1+\bar{p}_2 (1-\bar{p}_2)/n_2$, or (ii) $\bar{p} (1-\bar{p})/n_1+\bar{p} (1-\bar{p})/n_2$, where $\bar{p}$ is an appropriate estimator based on all the data.

A test based on either standard error will asymptotically have the correct significance level and are both reasonable tests, but the one with the pooled estimate of $p$ in the standard error is the more common.

If there are different sample sizes for the two proportions, you'd normally weight the sample proportions, not take a direct arithmetic average of the proportions; this makes more sense because you should be able to treat it like one larger sample under the null, and that's what taking the appropriately weighted average does.

Note that the two formulas will tend to give similar variance estimates unless the sample proportions are far from equal.

[I just did a simulation; for large sample sizes, the rejection rates were the same. For small sample sizes they differed because the normal approximation to different discrete distributions resulted in different actual significance levels. When you adjust the rejection rules to make the significance levels the same, the rejection rates look the same, which is to say there's little to prefer one or the other, though the type I error rate was better controlled (in that the nominal rate tended not to be exceeded) when the pooled estimate was used for standard errors. If you're happy with the actual rejection rates being different from desired, it seems there's little reason to prefer either one or the other.]

The formula you suggest that has $p(1-p)/2000$ under the square root is not a standard error of the difference in sample proportions, and so is not correct.


In response to question in comments:

I could not understand what you mean by "normal approximation to different discrete distributions" and how to adjust them

Sample proportions are ratios of integers; they have a discrete distribution. E.g. with a sample size of 10, the possible proportions are $0,0.1,0.2,...,1.0$. Because the two appropriate estimates of the standard error are based on the sample $p$ values, the standard error estimates also have a discrete distribution, and so does the usual two-sample proportions test statistic.

Differences of sample proportions are, as a result, discrete. The usual normal-based two sample proportions test is based a continuous approximation to that discrete distribution. But the two possible estimates of the standard error yield different discrete distributions at finite samples, which means the achievable significance levels are different.

By changing the rejection rules -- at least in the cases I played with -- I was able to get the same significance level, and when I did that, the power curves were the same. If I didn't do that, the difference in significance level caused a difference in power curve.

To obtain an appropriate rejection rule, I just tried a sequence of critical values until the simulated rejection rates at the null for one were as close as I could get to the desires significance level, and then I fiddled the other until I got the same significance level. (If it turned out not to be possible to make them exactly alike, there are other things that might be done to make them more comparable, but I don't want to labor the point.)

$\endgroup$
  • $\begingroup$ I could not understand what you mean by "normal approximation to different discrete distributions" and how to adjust them. $\endgroup$ – Val Nov 26 '13 at 9:54
  • $\begingroup$ Answer too long for a comment. I'll edit my answer. $\endgroup$ – Glen_b Nov 26 '13 at 10:18
  • $\begingroup$ Wait. By "significance level" do you mean the amount of standard deviations for the 95% confidence? I see that the mentor used the same 1.96 std deviations for 95% confidence in both cases. $\endgroup$ – Val Nov 26 '13 at 10:34
  • $\begingroup$ See discussion of significance levels here $\endgroup$ – Glen_b Nov 26 '13 at 10:42
  • $\begingroup$ I ask how do you "adjust them". I have the level of 95%, which corresponds to 1.96 standard deviations. I do not understand how constants can be adjusted. $\endgroup$ – Val Nov 26 '13 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.