Significance level example from "The Analysis of Experimental Data" paper

Question

The paper reads:

Fisher immediately realized that this argument fails because every possible result with the 6 pairs has probability (1/2)^6 = 1/64, so every result is significant at 5%. Fisher avoided this absurdity by saying that any outcome with just I W and 5 R’s, no matter where that W occurred, is equally suggestive of discriminatory powers and so should be included. There are 6 such possibilities, including the actual outcome, so the relevant probability for (a) above is 6(1/2)^6 = 6/64 = .094, so now the result is not significant at 5%.

I do not understand how 1/64 is significant at 5% but 6/64 is not. It makes more sense to me that bigger of two numbers would be deemed significant as it describes something that happens more often.

What is wrong with my reasoning?

Answer 1

Preliminary note: Based on the cited paragraph, I infer that you are talking about a test where you have six objects $X_1, ...., X_6 \sim \text{IID Bern}(\theta)$ (with outcomes labelled $W$ and $R$ respectively), and you are conducting the one-sided hypothesis test:

$$H_0: \theta = \tfrac{1}{2} \quad \quad \quad H_\text{A}: \theta < \tfrac{1}{2}.$$

You have observed data with one $W$s and five $R$s and you want to perform this one-sided test at the five-percent significance level. The cited paragraph suggests that "the argument" referred to is attempting to do this by calculating a p-value as the probability of the observed outcome under $H_0$.

---

"Significance" occurs when the p-value of the observed data is less than or equal to the chosen significance level. (Remember the mnemonic: "If p is low, the null must go!") The p-value is defined as the probability of observing data that is at least as conducive to the null hypothesis as what was actually observed, under the assumption that the null is true.

Since $p = 1/64 = 0.015625 < 0.05$, an observed outcome with this p-value would constitute significant evidence against the null hypothesis. In the paragraph you cite, the author is pointing out that a particular argument relating to calculating the p-value must be false, because it calculates the p-value as the probability of the observed data under the null hypothesis, and this leads to a "significant" result for every possible observable outcome. This is an absurdity, since it leads to a test that always rejects the null hypothesis, regardless of what is observed. (Note here that the p-value is not the probability of the observed data, under the null hypothesis. It is the probability of the observed data, or something else at least as conducive to the alternative hypothesis, ender the null. The argument referred to is counting the p-value too narrowly, hence the low value.)

In the paragraph you cite, this absurd result is contrasted with calculating the p-value by adding up the probability of outcomes that are as conducive to the alternative as the observed result. Actually, even here, there is an error in the paragraph you cite, since the p-value should include all six cases where we have $(1W,5R)$, but it should also include the single more extreme case $(0W,6R)$. So really, under this logic, and using the proper definition of a p-value based on a proper ordering of evidence, it should be $p=7/64=0.109375>0.05$. In any case, you can see that this result is not "significant" at the five-percent significance level.