I keep seeing this famous quote everywhere, but fail to understand the emphasized part every single time.
A man who ‘rejects’ a hypothesis provisionally, as a matter of habitual practice, when the significance is at the 1% level or higher, will certainly be mistaken in not more than 1% of such decisions. For when the hypothesis is correct he will be mistaken in just 1% of these cases, and when it is incorrect he will never be mistaken in rejection. [...] However, the calculation is absurdly academic, for in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas. It should not be forgotten that the cases chosen for applying a test are manifestly a highly selected set, and that the conditions of selection cannot be specified even for a single worker; nor that in the argument used it would clearly be illegitimate for one to choose the actual level of significance indicated by a particular trial as though it were his lifelong habit to use just this level.
(Statistical Methods and Scientific Inference, 1956, p. 42-45)
More specifically, I don't understand
- Why are the cases chosen for applying a test "highly selected"? Say you wonder if the average height of people within an area is less than 165cm, and decide to conduct a test. The standard procedure, as far as I know, is to draw random samples from the area and measure their height. How can this be highly selected?
- Suppose the cases are highly selected, but how is this related to the choice of the significance level? Consider again the example above, if your sampling method (what I suppose is what Fisher refers to as conditions of selection) is skewed and somehow favors tall people, then the whole research is ruined, and subjective determination of the significance level cannot save it.
- Actually, I don't even know what is "the actual level of significance indicated by a particular trial" referring to. Is it the $p$-value of that experiment, some preset value like the (in)famous 0.05, or something else?