EDIT:
If we look at Fisher's early (first?) description of the significance test we see that in the case of the lady tasting tea it was plausible that she could not distinguish the order in which tea/milk was added at all. Thus this was a logical null hypothesis. He also states the significance level is "open to the experimenter". However later he states that a null hypothesis that the death rates of two groups of animals are equal is also valid. In this case I think he simply missed the logical consequences of choosing an implausible null hypothesis as described by Paul Meehl in
Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It
Quoting Fisher:
A LADY declares that by tasting a cup of tea made with milk she can
discriminate whether the milk or the tea infusion was first added to
the cup. We will consider the problem of designing an experiment by
means of which this assertion can be tested.
...
It is open to the experimenter to be more or less exacting in respect
to the smallness of the probability he would require before he would
be willing to admit that his observations have demonstrated a positive
result. It is obvious that an experiment would be useless of which no
possible result would satisfy him. Thus, if he wishes to ignore
results having probabilities as high as 1 in 20 -the probabilities
being of course reckoned from the hypothesis that the phenomenon to be
demonstrated is in fact absent- then it would be useless for him to
experiment with only 3 cups of tea of each kind. For 3 objects can be
chosen out of 6 in only 20 ways, and therefore complete success in the
test would be achieved without sensory discrimination, i.e., by “pure
chance,” in an average of 5 trials out of 100. It is usual and
convenient for experimenters to take 5 per cent. as a standard level
of significance, in the sense that they are prepared to ignore all
results which fail to reach this standard, and, by this means, to
eliminate from further discussion the greater part of the fluctuations
which chance causes have introduced into their experimental results.
No such selection can eliminate the whole of the possible effects of
chance coincidence, and if we accept this convenient convention, and
agree that an event which would occur by chance only once in 70 trials
is decidedly “significant” in the statistical sense, we thereby admit
that no isolated experiment, however significant in itself, can
suffice for the experimental demonstration of any natural phenomenon;
for the “one chance in a million” will undoubtedly occur, with no less
and no more than its appropriate frequency, however surprised we may
be that it should occur to us. In order to assert that a natural
phenomenon is experimentally demonstrable we need, not an isolated
record, but a reliable method of procedure. In relation to the test of
significance, we may say that a phenomenon is experimentally
demonstrable when we know how to conduct an experiment which will
rarely fail to give us a statistically significant result.
...
It is evident that the null hypothesis must be exact, that is free
from vagueness and ambiguity, because it must supply the basis of the
“problem of distribution,” of which the test of significance is the
solution. A null hypothesis may, indeed, contain arbitrary elements,
and in more complicated cases often does so: as, for example, if it
should assert that the death-rates of two groups of animals are equal,
without specifying what these death-rates usually are. In such cases
it is evidently the equality rather than any particular values of the
death-rates that the experiment is designed to test, and possibly to
disprove.
I think a crucial point that others have missed in reading Fisher is that he never specifies a need to specify the significance level beforehand. A reasonable level could be chosen given the experimental conditions that unfolded. Also no strong conclusion about a natural phenomenon should be derived from a single study anyway, a single study could only provide evidence pointing towards the phenomenon's existence, thus suggesting the usefulness of further study. Thus the significance level could be arbitrary and based on convenience (since it could change) but it was not "binding".
While for Neyman-Pearson a single study could not provide any evidence for the existence of a phenomenon. The "significance" level was definitely non-arbitrary but it was binding if the pre-specified error rate of the decision making process were to be valid in the long run. Citing Neyman and Pearson the same as @MichaelLew in the above arxiv paper:
We are inclined to think that as far as a particular hypothesis is
concerned, no test based upon the theory of probability can by itself
provide any valuable evidence of the truth or falsehood of that
hypothesis.
But we may look at the purpose of tests from another
view-point. Without hoping to know whether each separate hypothesis is
true or false, we may search for rules to govern our behaviour with
regard to them, in following which we insure that, in the long run of
experience, we shall not be too often wrong.
The hybrid approach (which is being taught to researchers) uses a pre-specified arbitrary and binding significance level which is different from both. It neither adjusts the level due to experimental conditions nor involves cost-benefit analysis.
