# Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible" and hence the resulting "hybrid" is an "incoherent mishmash". I will provide a bibliography and some quotes below, but for now suffice it to say that there is a lot written about that in the wikipedia article on Statistical hypothesis testing. Here on CV, this point was repeatedly made by @Michael Lew (see here and here).

My question is: why are F and N-P approaches claimed to be incompatible and why is the hybrid claimed to be incoherent? Note that I read at least six anti-hybrid papers (see below), but still fail to understand the problem or the argument. Note also, that I am not suggesting to debate if F or N-P is a better approach; neither am I offering to discuss frequentist vs. Bayesian frameworks. Instead, the question is: accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

Here is how I understand the situation. Fisher's approach is to compute the $p$-value and take it as an evidence against the null hypothesis. The smaller the $p$, the more convincing the evidence. The researcher is supposed to combine this evidence with his background knowledge, decide if it is convincing enough, and proceed accordingly. (Note that Fisher's views changed over the years, but this is what he seems to have eventually converged to.) In contrast, Neyman-Pearson approach is to choose $\alpha$ ahead of time and then to check if $p\le\alpha$; if so, call it significant and reject the null hypothesis (here I omit large part of the N-P story that has no relevance for the current discussion). See also an excellent reply by @gung in When to use Fisher and Neyman-Pearson framework?

The hybrid approach is to compute the $p$-value, report it (implicitly assuming that the smaller the better), and also call the results significant if $p\le\alpha$ (usually $\alpha=0.05$) and nonsignificant otherwise. This is supposed to be incoherent. How can it be invalid to do two valid things simultaneously, beats me.

As particularly incoherent the anti-hybridists view the widespread practice of reporting $p$-values as $p<0.05$, $p<0.01$, or $p<0.001$ (or even $p\ll0.0001$), where always the strongest inequality is chosen. The argument seems to be that (a) the strength of evidence cannot be properly assessed as exact $p$ is not reported, and (b) people tend to interpret the right-hand number in the inequality as $\alpha$ and view it as type I error rate, and that is wrong. I fail to see a big problem here. First, reporting exact $p$ is certainly a better practice, but nobody really cares if $p$ is e.g. $0.02$ or $0.03$, so rounding it on a log scale is not soooo bad (and going below $\sim 0.0001$ does not make sense anyway, see How should tiny p-values be reported?). Second, if the consensus is to call everything below $0.05$ significant, then error rate will be $\alpha=0.05$ and $p \ne \alpha$, as @gung explains in Interpretation of p-value in hypothesis testing. Even though this is potentially a confusing issue, it does not strike me as more confusing than other issues in statistical testing (outside of the hybrid). Also, every reader can have her own favourite $\alpha$ in mind when reading a hybrid paper, and her own error rate as a consequence. So what is the big deal?

One of the reasons I want to ask this question is because it literally hurts to see how much of the wikipedia article on Statistical hypothesis testing is devoted to lambasting hybrid. Following Halpin & Stam, it claims that a a certain Lindquist is to blame (there is even a large scan of his textbook with "errors" highlighted in yellow), and of course the wiki article about Lindquist himself starts with the same accusation. But then, maybe I am missing something.

## Quotes

Gigerenzer: What has become institutionalized as inferential statistics in psychology is not Fisherian statistics. It is an incoherent mishmash of some of Fisher's ideas on one hand, and some of the ideas of Neyman and E. S. Pearson on the other. I refer to this blend as the "hybrid logic" of statistical inference.

Goodman: The [Neyman-Pearson] hypothesis test approach offered scientists a Faustian bargain -- a seemingly automatic way to limit the number of mistaken conclusions in the long run, but only by abandoning the ability to measure evidence [a la Fisher] and assess truth from a single experiment.

Hubbard & Bayarri: Classical statistical testing is an anonymous hybrid of the competing and frequently contradictory approaches [...]. In particular, there is a widespread failure to appreciate the incompatibility of Fisher's evidential $p$ value with the Type I error rate, $\alpha$, of Neyman-Pearson statistical orthodoxy. [...] As a prime example of the bewilderment arising from [this] mixing [...], consider the widely unappreciated fact that the former's $p$ value is incompatible with the Neyman-Pearson hypothesis test in which it has become embedded. [...] For example, Gibbons and Pratt [...] erroneously stated: "Reporting a P-value, whether exact or within an interval, in effect permits each individual to choose his own level of significance as the maximum tolerable probability of a Type I error."

Halpin & Stam: Lindquist's 1940 text was an original source of the hybridization of the Fisher and Neyman-Pearson approaches. [...] rather than adhering to any particular interpretation of statistical testing, psychologists have remained ambivalent about, and indeed largely unaware of, the conceptual difficulties implicated by the Fisher and Neyman-Pearson controversy.

Lew: What we have is a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence.

• @Momo: to you question about "what exactly is confusing" -- well, confusing is the frenzy of the anti-hybrid rhetoric. "Incoherent mishmash" are strong words, so I would like to see a pretty bad inconsistency. What you said about alternative hypothesis does not sound as such to me (in the garden variety case of $H_0: \mu=0$ the alternative is obviously $H_1: \mu \ne 0$, and I don't see much room for inconsistency), but if I am missing your point then maybe you would like to provide it as an answer. Aug 21, 2014 at 15:42
• Having just read Lew (and realizing I'd read it before, probably around 2006), I found it quite good, but I don't think it represents how I use p-values. My significance levels - on the rare occasions I use hypothesis testing at all* - are always up front, and where I have any control over sample size, after consideration of power, some consideration of the cost of the two error types and so on - essentially Neyman-Pearson. I still quote p-values, but not in the framework of Fisher's approach .... (ctd) Aug 22, 2014 at 2:26
• (ctd) ... * (I often steer people away from hypothesis testing - so often their actual questions are related to measuring effects, and are better answered by constructing intervals). The specific problem Lew raised for the 'hybrid' procedure applies to something I don't do and would tend to caution people against doing. If there are people really doing the mix of approaches he implies, the paper seems fine. The earlier discussion of the meaning of p-values and the history of the approaches seems excellent. Aug 22, 2014 at 2:28
• @Glen_b, Lew's historical overview is very nice and clear, I fully agree. My trouble is specifically with the hybrid issue (section "Which approach is most used?"). Certainly there are people doing what he describes there, i.e. reporting the strongest of p<.001, <.01, or <.05; I see it all the time in neuroscience. Consider one of the cases when you do use testing. You choose e.g. alpha=.05, and follow the NP framework. When you get p=.00011, is your certainty about H1 and your choice of wording going to be different from when you would get p=.049? If so, it is hybrid! If not, how come? Aug 22, 2014 at 11:02
• Thanks for the interesting question. Now, ~7 years later, have you come to a resolution? Would you mind posting it? May 22, 2021 at 17:31

I believe the papers, articles, posts e.t.c. that you diligently gathered, contain enough information and analysis as to where and why the two approaches differ. But being different does not mean being incompatible.

The problem with the "hybrid" is that it is a hybrid and not a synthesis, and this is why it is treated by many as a hybris, if you excuse the word-play.
Not being a synthesis, it does not attempt to combine the differences of the two approaches, and either create one unified and internally consistent approach, or keep both approaches in the scientific arsenal as complementary alternatives, in order to deal more effectively with the very complex world we try to analyze through Statistics (thankfully, this last thing is what appears to be happening with the other great civil war of the field, the frequentist-bayesian one).

The dissatisfaction with it I believe comes from the fact that it has indeed created misunderstandings in applying the statistical tools and interpreting the statistical results, mainly by scientists that are not statisticians, misunderstandings that can have possibly very serious and damaging effects (thinking about the field of medicine helps giving the issue its appropriate dramatic tone). This misapplication, is I believe, accepted widely as a fact-and in that sense, the "anti-hybrid" point of view can be considered as widespread (at least due to the consequences it had, if not for its methodological issues).

I see the evolution of the matter so far as a historical accident (but I don't have a $p$-value or a rejection region for my hypothesis), due to the unfortunate battle between the founders. Fisher and Neyman/Pearson have fought bitterly and publicly for decades over their approaches. This created the impression that here is a dichotomous matter: the one approach must be "right", and the other must be "wrong".

The hybrid emerged, I believe, out of the realization that no such easy answer existed, and that there were real-world phenomena to which the one approach is better suited than the other (see this post for such an example, according to me at least, where the Fisherian approach seems more suitable). But instead of keeping the two "separate and ready to act", they were rather superfluously patched together.

I offer a source which summarizes this "complementary alternative" approach: Spanos, A. (1999). Probability theory and statistical inference: econometric modeling with observational data. Cambridge University Press., ch. 14, especially Section 14.5, where after presenting formally and distinctly the two approaches, the author is in a position to point to their differences clearly, and also argue that they can be seen as complementary alternatives.

• (+1) I appreciate your comments and agree with many of them. But I am not sure what exactly you are referring to when you say that the hybrid "created misunderstandings" (and moreover, that this is "accepted widely as a fact"). Could you give some examples? To be an attack on the hybrid, it should be examples of misunderstandings that do not arise in either F or N-P approaches alone. Are you referring to the potential confusion between $p$ and $\alpha$ that I mentioned in my question, or to something else? Apart from that, I am already reading Section 14.5 in Spanos, thanks. Aug 21, 2014 at 16:07
• The obvious issue is indeed the $p-\alpha$ issue. More subtle and I believe more important, is the fact that the hybrid mixes the exploratory flavor of Fisher (which more over leaves the matter of decision to the researcher), with the more formal approach of N-P. So researchers approached the matter in a Fisherian spirit, but then claimed the strong "rejection/acceptance" weight of the N-P approach, which in principle gives more credibility to the conclusions. CONTD Aug 21, 2014 at 16:18
• CONTD For me, this is the "have your cake and eat it too" issue of the hybrid approach. For example, an N-P approach without power-test calculations should be unthinkable, but all the time we see test posed in the N-P framework, but no mention about power calculations. Aug 21, 2014 at 16:19
• Off topic, but... Since you are citing Aris Spanos, I wonder if you might be able to answer this question about his methodology? (I once asked the question to Aris Spanos directly, and he kindly put down some effort in answering it. Unfortunately, his answer was in the same language as his papers, thus it did not help me much.) Dec 17, 2019 at 18:23
• @RichardHardy I had a look -my problem is that I need to revisit concepts like "pre-test bias" in order to maybe be able to say something useful on your issue. Dec 18, 2019 at 20:58

My own take on my question is that there is nothing particularly incoherent in the hybrid (i.e. accepted) approach. But as I was not sure if I am maybe failing to comprehend the validity of the arguments presented in the anti-hybrid papers, I was happy to find the discussion published together with this paper:

Unfortunately, two replies published as a discussion were not formatted as separate articles and so cannot be properly cited. Still, I would like to quote from both of them:

Berk: The theme of Sections 2 and 3 seems to be that Fisher did not like what Neyman and Pearson did, and Neyman did not like what Fisher did, and therefore we should not do anything that combines the two approaches. There is no escaping the premise here, but the reasoning escapes me.

Carlton: the authors adamantly insist that most confusion stems from the marriage of Fisherian and Neyman-Pearsonian ideas, that such a marriage is a catastrophic error on the part of modern statisticians [...] [T]hey seem intent on establishing that P values and Type I errors cannot coexist in the same universe. It is unclear whether the authors have given any substantive reason why we cannot utter "p value" and "Type I error" in the same sentence. [...] The "fact" of their [F and NP] incompatibility comes as surprising news to me, as I'm sure it does to the thousands of qualified statisticians reading the article. The authors even seem to suggest that among the reasons statisticians should now divorce these two ideas is that Fisher and Neyman were not terribly fond of each other (or each other's philosophies on testing). I have always viewed our current practice, which integrates Fisher's and Neyman's philosophies and permits discussion of both P values and Type I errors -- though certainly not in parallel -- as one of our discipline's greater triumphs.

Both responses are very worth reading. There is also a rejoinder by the original authors, which does not sound convincing to me at all.

• It is one thing to co-exist, it is another for the one to be considered as the other. But indeed, this strand of anti-hybrid approach is in the spirit of "there can be no synthesis whatsoever" -which I strongly disagree with. But I don't see the current hybrid as a successful marriage. Aug 21, 2014 at 16:30
• @Livid, thanks for your comments, this is interesting, but I would like to refrain from further discussion here. I would rather encourage you to post a new answer, if you wish. But if you decide to do so, try to focus on the main issue, which is: what is so bad about "hybrid", as compared to both Fisher and N-P alone. You seem to hate the whole approach of significance testing, "nil null hypothesis", etc., but this is not what this question is about! Feb 14, 2015 at 23:33
• @Livid: Hmmm, can you actually clarify why you say that is is a distinguishing characteristic of the hybrid? What would the null be in pure Fisher or in pure NP? Say you have two groups and want to test for a significant difference ("nil null"). Can't one approach this situation with all three approaches: pure Fisher, pure NP, and hybrid? Feb 14, 2015 at 23:42
• @Livid, I understand your arguments against the nil null, I just think that this issue is orthogonal to the issue of hybrid. I have to refresh the anti-hybrid papers in memory, but as far as I remember their critique of the hybrid is not at all centered on the nil null. Instead, it is about combining Fisher and NP. Again, if you disagree with this, please consider posting an answer; for the moment, let's leave it at that. Feb 15, 2015 at 0:06
• A note to myself: I should incorporate into this answer some quotes from this paper: Lehmann 1992, The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two? Apr 21, 2015 at 17:12

I fear that a real response to this excellent question would require a full-length paper. However, here are a couple of points that are not present in either the question or the current answers.

1. The error rate 'belongs' to the procedure but the evidence 'belongs' to the experimental results. Thus it is possible with multi-stage procedures with sequential stopping rules to have a result with very strong evidence against the null hypothesis but a not significant hypothesis test result. That can be thought of as a strong incompatibility.

2. If you are interested in the incompatibilities, you should be interested in the underlying philosophies. The philosophical difficulty comes from a choice between compliance with the Likelihood Principle and compliance with the Repeated Sampling Principle. The LP says roughly that, given a statistical model, the evidence in a dataset relevant to the parameter of interest is completely contained in the relevant likelihood function. The RSP says that one should prefer tests that give error rates in the long run that equal their nominal values.

• J. O. Berger and R.L Wolpert's monograph "The Likelihood Principle" (2nd ed. 1988), is a calm, balanced, and good exposition of point 2., in my opinion. Aug 21, 2014 at 23:55
• Berger and Wolpert is indeed a good exposition, and authoritative too. However, I prefer the more practically directed and less mathemtatical book "Likelihood" by AWF Edwards. Still in print, I think. books.google.com.au/books/about/Likelihood.html?id=LL08AAAAIAAJ Aug 22, 2014 at 1:27
• @MichaelLew has explained that a valid use of p values is a summary of effect size. He has done a great thing by writing this paper: arxiv.org/abs/1311.0081 Feb 15, 2015 at 2:25
• @Livid The paper is v interesting, but for the new reader it's worth noting the following: the main idea, that p values 'index' (presumably: are in one to one relation with) likelihood functions, is generally understood to be false because there are cases where the same likelihood corresponds to different p-values depending on the sampling scheme. This issue is discussed a bit in the paper, but indexing is a very unusual position (which doesn't necessarily make it wrong, of course). Jan 9, 2016 at 14:15
• @conjugateprior The p-value to likelihood function indexing is one to one within the scope of a single statistical model, but the p-value from one model does not point to the same likelihood function as a numerically identical p-value from a different statistical model. cont... Jan 26 at 0:54

An often seen (and supposedly accepted) union (or better: "hybrid") between the two approaches is as follows:

1. Set a prespecified level $\alpha$ (0.05 say)
2. Then test your hypothesis, e.g. $H_o: \mu = 0$ vs. $H_1: \mu \ne 0$
3. State the p value and formulate your decision based on the level $\alpha$:

If the resulting p value is below $\alpha$, you could say

• "I reject $H_o$" or
• "I reject $H_o$" in favor of $H_1$" or
• "I am $100\% \cdot (1-\alpha)$ certain that $H_1$ holds"

If the p value is not small enough, you would say

• "I cannot reject $H_o$" or
• "I cannot reject $H_o$ in favor of $H_1$"

Here, aspects from Neyman-Pearson are:

• You decide something
• You have an alternative hypothesis at hand (although it is just the contrary of $H_o$)
• You know the type I error rate

Fisherian aspects are:

• You state the p value. Any reader has thus the possibility to use its own level (e.g. strictly correcting for multiple testing) for decision
• Basically, only the null hypothesis is required since the alternative is just the contrary
• You don't know the type II error rate. (But you could immediately get it for specific values of $\mu \ne 0$.)

While it is good to be aware of the discussion about the philosophical problems of Fisher's, NP's or this hybrid approach (as taught in almost religious frenzy by some), there are much more relevant issues in statistics to fight against:

• Asking uninformative questions (like binary yes/no questions instead of quantitative "how much" questions, i.e. using tests instead of confidence intervals)
• Data driven analysis methods that lead to biased results (stepwise regression, testing assumptions etc.)
• Choosing wrong tests or methods
• Misinterpreting results
• Using classic statistics for non-random samples
• (+1) This is a good description of the hybrid (and why exactly it is hybrid), but you did not explicitly say what your evaluation of it is. Do you agree that what you described is an "incoherent mishmash"? If so, why? Or do you think it is a reasonable procedure? If so, do the people claiming it is incoherent have a point, or are they simply wrong? Aug 21, 2014 at 15:35
• I often test hypotheses in exactly this manner... But there are other mish mashs I would not accept (e.g. not showing p values above $\alpha$) etc. Aug 21, 2014 at 15:50

accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

Short answer: the use of a nil (no difference, no correlation) null hypothesis irregardless of the context. Everything else is a "misuse" by people who have created myths for themselves about what the process can achieve. The myths arise from people attempting to reconcile their (sometimes appropriate) use of trust in authority and consensus heuristics with the inapplicability of the procedure to their problem.

As far as I know Gerd Gigerenzer came up with term "hybrid":

I asked the author [a distinguished statistical textbook author, whose book went through many editions, and whose name does not matter] why he removed the chapter on Bayes as well as the innocent sentence from all subsequent editions. “What made you present statistics as if it had only a single hammer, rather than a toolbox? Why did you mix Fisher’s and Neyman–Pearson’s theories into an inconsistent hybrid that every decent statistician would reject?”

To his credit, I should say that the author did not attempt to deny that he had produced the illusion that there is only one tool. But he let me know who was to blame for this. There were three culprits: his fellow researchers, the university administration, and his publisher. Most researchers, he argued, are not really interested in statistical thinking, but only in how to get their papers published [...]

The null ritual:

1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research hypothesis or of any alternative substantive hypotheses.

2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as $p < 0.05$, $p < 0.01$ , or $p < 0.001$ (whichever comes next to the obtained $p$-value).

3. Always perform this procedure.

Gigerenzer, G (November 2004). "Mindless statistics". The Journal of Socio-Economics 33 (5): 587–606. doi:10.1016/j.socec.2004.09.033.

Edit: And we should always need to mention, because the "hybrid" is so slippery and ill-defined, that using the nil null to get a p-value is perfectly fine as a way to compare effect sizes given different sample sizes. It is the "test" aspect that introduces the problem.

Edit 2: @amoeba A p-value can be fine as a summary statistic, in this case the nil null hypothesis is just an arbitrary landmark: http://arxiv.org/abs/1311.0081. However, as soon as you start trying to draw a conclusion or make a decision (ie "test" the null hypothesis) it stops making sense. In the comparing two groups example, we want to know how different two groups are and the various possible explanations there may be for differences of that magnitude and type.

The p value can be used as a summary statistic telling us the magnitude of the difference. However, using it to "disprove/reject" zero difference serves no purpose that I can tell. Also, I think many of these study designs that compare average measurements of living things at a single timepoint are misguided. We should want to observe how individual instances of the system change over time, then come up with a process that explains the pattern observed (including any group differences).

• +1, Thanks for your answer and for the link. It seems I haven't read this particular paper, I will take a look. As I said before, I was under impression that "nil null" is an issue orthogonal to the issue of "hybrid", but I guess I should re-read Gigerenzer's writings to check that. Will try to find time in the following days. Apart from that: could you please clarify your last paragraph ("edit")? Did I understand correctly that you meant that having a nil null when comparing two effect sizes is okay, but having a nil null when comparing an effect size to zero is not okay? Feb 15, 2015 at 14:19

I see that those with more expertise than myself have provided answers, but I think my answer has the potential to add something additional, so I'll offer this as one other layman's perspective.

Is the hybrid approach incoherent?  I'd say it depends on whether or not the researcher ends up acting inconsistently with the rules that they started out with: specifically the yes/no rule that comes into play with the setting of an alpha value.

## Incoherent

Start with Neyman-Pearson.  Researcher sets alpha=0.05, runs the experiment, calculates p=0.052.  Researcher looks at that p-value and, using Fisherian inference (often implicitly), considers the result to be sufficiently incompatible with the test hypothesis that they will still claim "something" is going on.  The result is somehow "good enough" even though the p-value was greater than the alpha value.  Often this is paired with language such as "nearly significant" or "trending towards significance" or some wording along those lines.

However, setting an alpha value before running the experiment means that one has chosen the approach of Neyman-Pearson inductive behavior.  Choosing to ignore that alpha value after calculating the p-value, and thus claiming something is still somehow interesting, undermines the entire approach that one started with.  If a researcher starts down Path A (Neyman-Pearson), but then jumps across to another path (Fisher) once they don't like the path they are on, I consider that incoherent.  They are not being consistent with the (implied) rules that they started with.

## Coherent (possibly)

Start with N-P.  Researcher sets alpha=0.05, runs the experiment, calculates p=0.0014.  Researcher observes that p < alpha, and thus rejects the test hypothesis (typically no effect null) and accepts the alternative hypothesis (the effect is real).  At this point the researcher, in addition to deciding to treat the outcome as a real effect (N-P), decides to infer (Fisher) that the experiment provides very strong evidence that the effect is real.  They have added nuance to the approach they started with, but have not contradicted the rules set in place by choosing an alpha value at the beginning.

## Summary

If one starts by choosing an alpha value, then one has decided to take the Neyman-Pearson path and follow the rules for that approach.  If they, at some point, violate those rules using Fisherian inference as the justification, then they have acted inconsistently/incoherently.

I suppose one could go a step further and declare that because it is possible to use the hybrid incoherently, therefore the approach is inherently incoherent, but that seems to be getting deeper into the philosophical aspects, which I don't consider myself qualified to even offer an opinion on.

Hat tip to Michael Lew.  His 2006 article helped me understand these issues better than any other resource.