# Are Fisher's tests of significance mathematically correct?

I think of Fisher's approach like this:

We choose a test statistic whose distribution is calculated under H0. H0 being a simple hypothesis of preference

We break down the distribution according to significance thresholds

The significance thresholds are defined by what we consider to be an extreme result, that is to say a result which would happen very little and therefore would put us in doubt on the veracity of H0.

We calculate the p-value which corresponds to the probability of obtaining a result at least as extreme as the realisation of the test statistic under H0. If the p-value is lower than the significance threshold we reject H0 otherwise we do not reject H0.

It is often said that Neyman-Pearson rigorously mathematized Fisher's significance tests. They therefore added the concept of alternative hypothesis and second-kind error. However, Fisher has long insisted on the reverse. I would then like to know if this is true. Can Fisher significance tests exist mathematically without an alternative hypothesis? (Other than by simple equivalence). If no? What is the mathematical use of the alternative hypothesis (Other than devine the second-kind error)? Because it apparently has an asymmetric role with H0 in the Neyman-Pearson tests. If yes ? So how did Fisher plan to deal with the concept of effect size and the impact of sample size on p value?

Sorry if my english is not very good, I am a french student.

• Jul 26, 2021 at 10:20

The alternative hypothesis in the Neyman-Pearson approach is introduced in order to make optimality statements possible. Tests used and interpreted according to Fisher should be correct in the sense that the p-value is uniformly distributed, meaning that any desired test level is kept. In order to decide what test (statistic) to use in what situation, Fisher would use statistical and subject matter insight, and the mathematics is just used in order to show that the test works correctly under $$H_0$$ (in some cases asymptotically/approximately).

The Neyman-Pearson approach allows to make mathematical statements of the kind that "among all tests that are correct for a given problem under the $$H_0$$, test X is optimal in the sense that it has the optimal power against all alternatives in set S". This can be used to decide what test to use and for what reason.

Fisher doesn't have a mathematical theory for making this decision. This doesn't mean that "Fisher's tests mathematically don't exist" or "are not correct". They are well defined and exist as mathematical objects. He just doesn't have optimality theory.

Now the point that Fisher has sometimes made against the Neyman-Pearson approach is that the optimality theory may be problematic because in reality model assumptions for the alternative are not fulfilled, and a test that is optimal against certain specific alternatives may be bad in some other situations in which the truth (assuming that the alternative holds) does not have the specified form. The thing is that formalising something mathematically always means to idealise it and to constrain it to some extent, and one can always argue that in reality something may go on that is not properly captured by the mathematical formalism. A mathematical optimality theory makes sense to the extent that what is optimised is really desirable, and the model situation considered is an appropriate model for the real situation in question. If this is not the case, the mathematical optimality result is not helpful. (Note that Fisher still makes assumptions about the $$H_0$$; his argument is that he doesn't want to specify in which specific way it could be violated.)

However personally I think that Neyman-Pearson optimality theory is fine as a mathematical theory, and also fine for picking a test in situations in which the model is appropriate, having in mind that it has these limitations and will not always guide us well. It just provides some information that can be valuable, even though it isn't always. So Fisher has a point, but I think that the consequence is rather to be careful with the theory rather than Fisher's polemical stance (that probably had something to do with issues between the persons) that it is useless to derive such a theory in the first place.

So ultimately the question has to do with the tension between reality and mathematics. We can always learn something doing mathematics, but there is a danger in trusting the mathematics too much when applying it to reality. To say that Neyman-Pearson is valid and Fisher's stance is invalid because Neyman-Pearson have some theory that Fisher hasn't is wrong, because Fisher's testing makes well defined sense (and is not in any mathematical way incorrect) also without that theory. On the other hand, we should still appreciate the value of Neyman-Pearson's theory when interpreted with enough care (and occasionally ignoring it when reality does not play ball).

• Hello, Thank you for your answer. It is very constructive and allows me to see more clearly. So do you know how to deal with the effect size and the impact of the sample size on the p-value in Fisher theory? Thank you Jul 26, 2021 at 17:43
• Sorry, you better make this another question. One can obviously do theory and simulations about how sample size and effect size affect p-values, however one might argue that this is not " Fisherian". Personally I don't care about that. To me it makes sense and I don't mind whether Fisher didn't like it. (As I hinted at in my answer, I think Fisher became at some point overly negative about Neyman and Pearson, and I just don't follow him there.) Jul 26, 2021 at 22:50