0
$\begingroup$

My question is about the interpretation of p-value. Under the null hypothesis the p-value is uniformly distributed. Then why does a test statistic landing in the far ends together comprising 5% of probability support ruling out the null hypothesis at 5% confidence when we don't say the same about any other interval.

To make it more concrete take the null hypothesis of a standard normal distribution and divide the number line into 40 equi-probability (under the standard normal) non-overlapping intervals. Why is the test statistic landing in one of the two extremes (with a total of 5% chance under the null hypothesis) rule out the null hypothesis at 5% confidence interval? Why don't we say the same about a combination of any of the $\phantom{a}^{40} C_2$ intervals?

For comparison think of a 40-faced dice. We want to check the null hypothesis of the dice being fare. If we get 1 or 40 would we say at 5% confidence we can say the null hypothesis is ruled out?

$\endgroup$
  • $\begingroup$ Fisher would say "extreme" was the key word here. More precisely, you want to use the rejection region which has the most statistical power. You want to minimise Type II errors where you fail to reject a false null hypothesis $\endgroup$ – Henry Nov 8 '16 at 10:28
  • $\begingroup$ I think I can see what you are saying for the Gaussian but what about the dice example? $\endgroup$ – Borun Chowdhury Nov 8 '16 at 10:45
  • $\begingroup$ For your example of a single roll of a $40$-sided die, it might depend on your null hypothesis. One which said "all faces are equally likely" needs more than one roll to have any information which might possibly be seen as "extreme", while one which said "the expected value of a roll is $20.5$" could be tested with a single roll in the way you suggest. This is an example of why some statisticians do not like frequentist hypothesis tests. $\endgroup$ – Henry Nov 8 '16 at 11:35
  • $\begingroup$ Thanks. Just to be sure for the second null hypothesis you mention getting a 40 or 1 would rule out the null hypothesis of a expected value of 20.5 at 5% confidence interval according to the methods of frequentist hypothesis testing? $\endgroup$ – Borun Chowdhury Nov 9 '16 at 11:29
  • $\begingroup$ Not "rule out", but "reject the null hypothesis at a $5\%$ significance level" in the knowledge that when the null hypothesis is true you may have a $5\%$ probability of erroneously rejecting it $\endgroup$ – Henry Nov 9 '16 at 11:58
3
$\begingroup$

Choosing any given 5% of the null distribution will leave you with a valid test (in that it will have a 5% significance level) but most people are somewhat concerned that their test not only reject the null that small proportion of the time when $H_0$ is true but that it also have better than a 5% chance to reject $H_0$ when it's false.

As a result, people try to consider power in some way when choosing a rejection region (i.e. which 5% - or whatever other $\alpha$ - of the null distribution to reject for). For example:

  1. If you have some test statistic and you have a test with some particular kind of alternative you'd normally choose the 5% (the proportion $\alpha$) of the null distribution that's most consistent with the alternative to be the 5% you reject. For example, if you had a permutation test for a difference in means you'd choose the 5% that had the biggest (absolute) difference in means. So when the null was more wrong you would have a better chance to be in the rejection region. This is the general consideration.

    In cases where you have a broader set of alternatives (such as when testing goodness of fit) you would choose a test statistic that in some general sense picks up (/measures) the kinds of alternatives you care about (typically, that it measures discrepancy from the ideal null case -- lack of fit -- in some way and so tends to be "large" when the null is false) and then you reject for the most discrepant 5%.

  2. If you're doing a likelihood ratio test you choose the 5% of the null distribution where the likelihood ratio $\mathcal{L}_0/\mathcal{L}_1$ is smallest.

  3. If you're basing a test off the likelihood itself (as with a Fisher exact test, for example), you'd choose the 5% of the null distribution with the lowest likelihood.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.