What does Fisher mean by this quote?

Question

I keep seeing this famous quote everywhere, but fail to understand the emphasized part every single time.

A man who ‘rejects’ a hypothesis provisionally, as a matter of habitual practice, when the significance is at the 1% level or higher, will certainly be mistaken in not more than 1% of such decisions. For when the hypothesis is correct he will be mistaken in just 1% of these cases, and when it is incorrect he will never be mistaken in rejection. [...] However, the calculation is absurdly academic, for in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas. It should not be forgotten that the cases chosen for applying a test are manifestly a highly selected set, and that the conditions of selection cannot be specified even for a single worker; nor that in the argument used it would clearly be illegitimate for one to choose the actual level of significance indicated by a particular trial as though it were his lifelong habit to use just this level.

(Statistical Methods and Scientific Inference, 1956, p. 42-45)

More specifically, I don't understand

1. Why are the cases chosen for applying a test "highly selected"? Say you wonder if the average height of people within an area is less than 165cm, and decide to conduct a test. The standard procedure, as far as I know, is to draw random samples from the area and measure their height. How can this be highly selected?
2. Suppose the cases are highly selected, but how is this related to the choice of the significance level? Consider again the example above, if your sampling method (what I suppose is what Fisher refers to as conditions of selection) is skewed and somehow favors tall people, then the whole research is ruined, and subjective determination of the significance level cannot save it.
3. Actually, I don't even know what is "the actual level of significance indicated by a particular trial" referring to. Is it the $p$-value of that experiment, some preset value like the (in)famous 0.05, or something else?

Answer 1

Here is my paraphrase of what Fisher says in your bolded quote. It should not be forgotten that quite a lot goes into choosing what hypothesis to test, so much so that even for a single person's decision, you could not specify it all. It also should not be forgotten that, for reasons stated above, you cannot decide on a particular trial's significance level always the same way, as a life long habit.

1. A scientific hypothesis is selected as worth testing against many other competing hypotheses because of the biases of the researcher and their current state of knowledge. The hypotheses are "highly selected", not the samples; the hypotheses are the cases where we apply tests.

2. The selection process of the hypotheses affects our significance level. If we are very sure of a hypothesis, that should make the significance level less stringent to satisfy ourselves. If we are unsure there is higher burden of proof. Other factors come into play as well, such as Type I error being worse than Type II in drug trials.

3. I think when he says "indicated by" he simply means "chosen for". Yes, it is a preset value where we reject the hypothesis if the p-value is more extreme.

Answer 2

The cases to which Fisher is referring are not observations but tests. That is, we select hypotheses to test. We don't just test random hypotheses - we base them on observation, the literature, scientific theories and so on.

If you did test random hypotheses, then the number of times you are mistaken (in the first sentence of your quote) would be 1% (or whatever value is chosen). E.g. if we tested hypotheses like

• The parity of a person's social security number is related to his IQ

• Blond haired people throw Frisbees better than dark haired people

• The time to getting an answer on Cross Validated is related to the number of syllables in your first name.

And tested a whole bunch of them at 1%, we would reject the null about 1% of the time, and do so incorrectly. (Unless, of course, I am on to something with the above nonsense).

I did once see an article about hair color and Frisbee throwing - and it found a difference! So, I call this sort of thing "Frisbee research".

But the part I like the best from the quote is this:

for in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.

He must be spinning in his grave.

Answer 3

Trying to see the background of the quote I came to a version of the book (I am not sure which is which version) that has a slightly different quote

https://archive.org/details/in.ernet.dli.2015.134555/page/n47

The attempts that have been made to explain the cogency of tests of significance in scientific research, by reference to hypothetical frequencies of possible statements, based on them, being right or wrong, thus seem to miss the essential nature of such tests. A man who "rejects" a hypothesis provisionally, as a matter of habitual practice, when the significance is at the 1% level or higher, will certainly be mistaken in not more than 1% of such decisions. For when the hypothesis is correct he will be mistaken in just 1% of these cases, and when it is incorrect he will never be mistaken in rejection. This inequality statement can therefore be made. However, the calculation is absurdly academic, for in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas. Further, the calculation is based solely on a hypothesis, which, in the light of the evidence, is often not believed to be true at all, so that the actual probability of erroneous decision, supposing such a phrase to have any meaning, may be much less than the frequency specifying the level of significance. To a practical man, also, who rejects a hypothesis, it is, of course, a matter of indifference with what probability he might be led to accept the hypothesis falsely, for in his case he is not accepting it.

This seems to me a criticism to use the mathematical expression of rejection possibilities, type I errors, as some rigorous argument. Those expressions are often not a good expression for what is relevant and neither are they rigorous.

1. Why are the cases chosen for applying a test "highly selected"?

   This seems to relate to the sentence

   Further, the calculation is based solely on a hypothesis, which, in the light of the evidence, is often not believed to be true at all

   We are not indifferent towards the hypothesis that is being tested, and often a hypothesis that is being tested is not believed to be true.

2. how is this related to the choice of the significance level?

   This relates to

   so that the actual probability of erroneous decision, supposing such a phrase to have any meaning, may be much less than the frequency specifying the level of significance

   The p-value is just the frequency of making a mistake when the null-hypothesis is true. But the actual frequency of making a mistake will be different (lower).

3. what is "the actual level of significance indicated by a particular trial" referring to

   I believe that this part refers to some sort of p-value hacking. Changing the significance level, alpha, after the observations have occurred in order to match the observed p-value, and pretend that this was the cut-off value all along from the beginning.