1
$\begingroup$

In reading through the literature on z-tests for difference in proportions, I often see it stated that the null hypothesis only tests whether p1 = p2, not whether p1 = p2 = any specific value for the pooled proportion. But in fact when we use the overall/pooled proportion from our sample to estimate the population proportion to calculate our test statistic, are we not actually making a concrete assumption about a specific value that p1 and p2 are both equal to?

As an illustration, suppose a researcher is interested in evaluating whether men or women in a large city are more likely to own a pet. The researcher performs separate, independent random samples of 20 men and 20 women and records whether they own a pet or not. Now lets say the results are that 8 of the 20 men own pets, and 12 of the 20 women own pets. The overall (pooled) proportion of pet owners from this exercise is 20/40 or 0.5. By using 0.5 to calculate our test statistic, aren't we really testing the probability that p1 = p2 = 0.5? There is nothing in the standard calculation of the test statistic which says anything about the probability that p1 = p2 = 0.2, or 0.8, or any other value other than 0.5.

It seems to me that if you wanted to really test the probability p1 = p2 = any specific value, you would need to run separate tests for specific different possible p1=p2 values, and then weight the outcomes by the likelihood of these different possible prior probabilities. But that begs the question of what the distribution of prior probabilities looks like....

But really rather than getting into a long discussion on a much more complex analysis involving estimating alternative prior probability distributions, I really am looking for confirmation that the more common approach of using a z-test for difference in proportions is actually testing the validity of a null hypothesis that p1 = p2 = the specific value as determined by the pooled sample proportion - despite what I have commonly encountered in textbooks and online explanations which claim that no specific value for p1 = p2 is being tested.

Improve this question
$\endgroup$
3
  • $\begingroup$ An astute question. See also On Fisher's exact test: What test would have been appropriate if the lady hadn't known the number of milk-first cups? for some discussion of exact tests. Nuisance parameters can be a nuisance. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    May 26, 2016 at 19:13
  • $\begingroup$ Thanks for your comment! Since first learning about the FET, I have always felt it to be a very exact answer in search of proper question. I have tried to calculate a p-values more directly by simply looking at all the possible discrete outcomes for the product of two binomials, and summing all outcomes which have an equal or lower probability of occurring compared to the specific outcome observed in the 2 independent samples. After all, isn't the purpose of hypothesis testing to see just how rare the outcome we have observed must be if the null is true? $\endgroup$
    – user221943
    May 26, 2016 at 20:00
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    May 27, 2016 at 9:54

1 Answer 1

Reset to default
2
$\begingroup$

The textbooks are right and you test $p_1=p_2$ vs. them not being equal. The standard test statistic is a quantity., for which the distribution is asymptotically known under the null hypothesis, which is the basis for using it. It so happens to involve the observed proportions (unsurprisingly, really), but the value of $p_1$ is just a nuissance parameter.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you - I have accepted your answer but I must admit the intuition for this still escapes me. I understand that the observed pooled proportion is the Most Likely Estimate (MLE) and therefore is certainly superior to using any other alternative. But this concept of the value of p1 and p2 being a nuisance parameter that doesn't really impact our hypothesis test is counter-intuitive for me. There is certainly some probability of the true proportion being say 30% instead of 50%, and using 30% instead of 50% would certainly change test-statistic and the corresponding p-value, no? $\endgroup$
    – user221943
    May 26, 2016 at 18:44
  • $\begingroup$ Using a different number would give a different test statistics, but under repeated sampling using this formula gives a level $\alpha$ test. Or frame it as a logistic regression, where we assume $Y_i \sim \text{Bin}(n_i, p)$ and $\text{logit}(p) = \text{logit}(p_1) + 1{i=2} \times \beta$. If you do a hypothesis test for the log-odds ratio $\beta$, you are not doing a hypothesis test for $\text{logit}(p_1)$ and you could commit a separate type I error for it. Another thing is that $p_1 = p_2 = \text{sample proportion}$ is not an up-front hypothesis (depends on the sample proportion!). $\endgroup$
    – Björn
    May 26, 2016 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.