How does Fisher calculate his $p$-value?

Question

After reading a lot of great answers on the topic of Fisherian versus Neyman & Pearson, I still cannot understand how Fisher carries out his test.

Here is my understanding of his workflow:

1. Ask a question.
2. Propose a null hypothesis based on the question.
3. Do some experiments, and collect some data.
4. Assuming the null is true, calculate the $p$-value.
5. Report the exact $p$-value without comparing it with any explicit criteria.
6. Take a look at your $p$:
   1. If you subjectively think it's too small, discard the data at your hand and go to the outermost (2) by proposing another null hypothesis.
   2. If you subjectively think large it's enough, discard the data at your hand, do another experiment, collect some new data, and find better ways to test your theory on them.

(From my understanding, the data used in the NHST stage cannot be used in further steps, regardless of whether we are following Fisher or Neyman & Pearson.)

I'm not sure if I'm correct, but there is a problem even if I am. Conventionally, the $p$-value is defined as

the probability of obtaining a test statistic at least as extreme as the actual sample value obtained given that the null hypothesis is true.

OK, but how do you define extreme without an alternative hypothesis? Everyone is a bit loose with their language when it comes to this issue, so while illustrative examples don't hurt, please don't post answers containing only examples. I am essentially asking for a high-level description of how Fisher chooses his critical region.

Note that this is not a problem for the Neyman-Pearson approach, because they didn't even mention the $p$-value in their 1933 paper that proposes the Neyman–Pearson lemma, so we can define Neyman-Pearson hypothesis testing in terms of critical regions, and then establish a bijection between the $p$-value and the critical regions. On the other hand, Fisher doesn't seem to have a single paper this clearly documents his way, and I'm a little confused.

Answer 1

Fisher's approach, in a fully parametric framework, was to reduce the data $X$ to a (one-dimensional) statistic sufficient, or conditionally sufficient, for the parameter of interest $\theta$, & to base inference on its distribution under the null hypothesis $\theta=\theta_0$. Typically he used the (or a) maximum-likelihood estimate $\hat\theta(X)$ (in any case the MLE, when unique, will be a one-to-one function of any one-dimensional sufficient statistic when there is one); though I don't recall any explicit discussion, viewing the maximum-likelihood estimate $\hat\theta(X_1)$ as more extreme than $\hat\theta(X_2)$ because it's further away from $\theta_0$ in the same direction follows naturally enough. See Fisher (1934), Proc. Royal Soc. Lond. A, 144 ,"Two New Properties of Mathematical Likelihood", § 2.6, for his emphasis on the connection between (maximum-likelihood) estimation & significance testing.

He doesn't seem to have given a great deal of thought to the calculation of p-values for two-tailed tests (at least for test statistics having discrete distributions). Yates (1984), JRSS A, 147, "Tests of Significance for 2x2 Contingency Tables", p. 444, quotes Fisher's (1946) reply to a letter from D.J Finney asking about two-tailed p-values for Fisher's Exact Test:

I believe I can defend the simple solution of doubling the total probability, not because it corresponds to any discrete subdivision of cases of the other tail, but because it corresponds with halving the probability, supposedly chosen in advance, with which the one observed is to be compared. [...] How does this strike you?

On the face of it this argument belongs more to the Neyman – Pearson approach.

Fisher (1973), Statistical Methods & Scientific Inference, pp 49 – 50, draws a distinction between testing a "general hypothesis"—a model—as a whole, & testing for a particular value of one of its parameters. In the latter case he reiterates the approach above; in the former his advice is this:

In choosing the grounds upon which a general hypothesis should be rejected, personal judgement may & should properly be exercised. The experimenter will rightly consider all points on which, in the light of current knowledge, the hypothesis may be imperfectly accurate, & will select tests, so far as possible, sensitive to these possible faults, rather than to others.

Which doesn't seem poles apart from the approach of stipulating an alternative hypothesis precisely & basing your choice of test statistic on considerations of power.