After reading a lot of great answers on the topic of Fisherian versus Neyman & Pearson, I still cannot understand how Fisher carries out his test.
Here is my understanding of his workflow:
- Ask a question.
- Propose a null hypothesis based on the question.
- Do some experiments, and collect some data.
- Assuming the null is true, calculate the $p$-value.
- Report the exact $p$-value without comparing it with any explicit criteria.
- Take a look at your $p$:
- If you subjectively think it's too small, discard the data at your hand and go to the outermost (2) by proposing another null hypothesis.
- If you subjectively think large it's enough, discard the data at your hand, do another experiment, collect some new data, and find better ways to test your theory on them.
(From my understanding, the data used in the NHST stage cannot be used in further steps, regardless of whether we are following Fisher or Neyman & Pearson.)
I'm not sure if I'm correct, but there is a problem even if I am. Conventionally, the $p$-value is defined as
the probability of obtaining a test statistic at least as extreme as the actual sample value obtained given that the null hypothesis is true.
OK, but how do you define extreme without an alternative hypothesis? Everyone is a bit loose with their language when it comes to this issue, so while illustrative examples don't hurt, please don't post answers containing only examples. I am essentially asking for a high-level description of how Fisher chooses his critical region.
Note that this is not a problem for the Neyman-Pearson approach, because they didn't even mention the $p$-value in their 1933 paper that proposes the Neyman–Pearson lemma, so we can define Neyman-Pearson hypothesis testing in terms of critical regions, and then establish a bijection between the $p$-value and the critical regions. On the other hand, Fisher doesn't seem to have a single paper this clearly documents his way, and I'm a little confused.