Why are we considering all extreme events in a hypothesis test?

Question

Why do we consider the more extreme possible observations of the test statistic when hypothesis testing? For example, I tried having a read of the tea-testing scenario, where a man named Fisher tries to test to see if Muriel has "discriminatory powers" in guessing whether a given cup of tea had milk put in first or last, as she so claimed. She guessed 5 out of 6 right.

I got stuck here (the bold part):

Fisher realised that this doesn't work; every possible outcome with one wrong pair was equally suggestive of discriminatory powers. The relevant probability for situation (a), above, is therefore 6(0.5)^6 = 0.094 (or 6/64) which now is not significant at a significance level of 5%. To overcome this Fisher argued that if 1 error in 6 is considered evidence of discriminatory powers then so is no errors i.e. outcomes that more strongly indicate discriminatory powers than the one observed should be included when calculating the p-value. This resulted in the following amendment to the...

Why are we even considering the chance of no errors? The observed value of $X$ (the number of errors she got) was 1 out of 6 times. Why aren't we saying "Hey, $X \sim B(6, 0.5)$. Let's find $P(X = 1)$"? Why are we saying "Lets find $P(X \leq 1)$"? She never got 0 errors.

This is a problem I'm having in general with hypothesis testing. Why aren't we just sticking to what we observed?

Answer 1

The reason for considering "more extreme" observations is that we need to "rank" your observed data in terms of extremity assuming a true null hypothesis. Why? Because if the data you observe are very aberrant under the assumption of a true null hypothesis, then you might consider rejecting that null hypothesis.

How do we quantify how "aberrant" your data are under the null? It's about relativity: assuming a true null hypothesis, relative to all possible datasets we could have observed, where does your observed data "rank" on the extremity scale? This is what the p-value tells you. If your p-value is 0.0001, you are saying that assuming the null hypothesis is true, if we were to repeat the gathering-of-the-data process many many many times, only about 1 in 10,000 times would we collect a dataset that is as extreme or more extreme than yours. So your dataset is "very extreme" with respect to the null (or at least I would say 1 in 10,000 is pretty extreme in most scenarios).

Now you ask yourself: given what I have observed (i.e. a small p-value), do I feel more comfortable saying that the null hypothesis really is true, and that I got an extreme dataset by random chance? Or do I feel more comfortable positing that the null hypothesis is false, and under a more appropriate hypothesis, these data wouldn't look so extreme.

Answer 2

The null hypothesis $H_0$ is that Muriel has no ability to detect whether milk or tea was added first. If $H_0$ is true then the number of correct responses out of five is $X \sim\mathsf{Binom}(n=5, p =1/2).$ So $P(X = 5) = 1/32.$ Also, $P(X = 4) = 5/43.$

In testing a hypothesis we can never say for sure whether $H_0$ is true. We can only talk about the data. If Muriel gets five answers out of five correct, we can say that's a 'rare' event---probability $1/32.$ We can't say for sure that $H_0$ is false, but we can be doubtful because believing $H_0$ gives us a rare event that's relatively hard to believe. Maybe Muriel really can tell the difference whether milk or tea was added first.

Somewhere along the line, $1/20 = 5\%$ got established as the borderline between 'too rare to believe' and 'not so surprising'. If Muriel gets only four of five correct then $H_0$ leads us to probability $5/32 > 5\%,$ which is on the 'not so surprising' side. (Maybe it's more fair to say $P(X \ge 5) = 6/32 = 3/16.)$

Maybe Muriel can tell the difference some of the time--- if she 'got good sleep' the night before, if the tea is her 'favorite brand', if it's a 'beautiful sunny day', etc. But, with anything less than five out of five, her ability isn't good enough or consistent enough to convince us of her general reliability as a judge of how tea is prepared.

Notes:

(1) Fisher made many useful contributions to statistics, but some people feel that he didn't always give the the clearest, most convincing arguments in favor of his ideas. What is above is my version of the explanation of testing this hypothesis. Maybe it's a little different from Fisher's.

(2) If Muriel felt up to judging ten cups of tea in sequence, then the probability of getting 9 or 10 correct would be about $0.002 < 5\%$ and then we would reject $H_0.$ But not if she got only 8 correct, which would have a probability of 8 or more correct of $0.055 > 5\%$ and so not be 'significant' at the 5% level. (Whether Muriel would claim to be able to judge ten cups in a row without confusion or 'taste fatigue' is another issue.)