# Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible" and hence the resulting "hybrid" is an "incoherent mishmash". I will provide a bibliography and some quotes below, but for now suffice it to say that there is a lot written about that in the wikipedia article on Statistical hypothesis testing. Here on CV, this point was repeatedly made by @Michael Lew (see here and here).

My question is: why are F and N-P approaches claimed to be incompatible and why is the hybrid claimed to be incoherent? Note that I read at least six anti-hybrid papers (see below), but still fail to understand the problem or the argument. Note also, that I am not suggesting to debate if F or N-P is a better approach; neither am I offering to discuss frequentist vs. Bayesian frameworks. Instead, the question is: accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

Here is how I understand the situation. Fisher's approach is to compute the $p$-value and take it as an evidence against the null hypothesis. The smaller the $p$, the more convincing the evidence. The researcher is supposed to combine this evidence with his background knowledge, decide if it is convincing enough, and proceed accordingly. (Note that Fisher's views changed over the years, but this is what he seems to have eventually converged to.) In contrast, Neyman-Pearson approach is to choose $\alpha$ ahead of time and then to check if $p\le\alpha$; if so, call it significant and reject the null hypothesis (here I omit large part of the N-P story that has no relevance for the current discussion). See also an excellent reply by @gung in When to use Fisher and Neyman-Pearson framework?

The hybrid approach is to compute the $p$-value, report it (implicitly assuming that the smaller the better), and also call the results significant if $p\le\alpha$ (usually $\alpha=0.05$) and nonsignificant otherwise. This is supposed to be incoherent. How can it be invalid to do two valid things simultaneously, beats me.

As particularly incoherent the anti-hybridists view the widespread practice of reporting $p$-values as $p<0.05$, $p<0.01$, or $p<0.001$ (or even $p\ll0.0001$), where always the strongest inequality is chosen. The argument seems to be that (a) the strength of evidence cannot be properly assessed as exact $p$ is not reported, and (b) people tend to interpret the right-hand number in the inequality as $\alpha$ and view it as type I error rate, and that is wrong. I fail to see a big problem here. First, reporting exact $p$ is certainly a better practice, but nobody really cares if $p$ is e.g. $0.02$ or $0.03$, so rounding it on a log scale is not soooo bad (and going below $\sim 0.0001$ does not make sense anyway, see How should tiny p-values be reported?). Second, if the consensus is to call everything below $0.05$ significant, then error rate will be $\alpha=0.05$ and $p \ne \alpha$, as @gung explains in Interpretation of p-value in hypothesis testing. Even though this is potentially a confusing issue, it does not strike me as more confusing than other issues in statistical testing (outside of the hybrid). Also, every reader can have her own favourite $\alpha$ in mind when reading a hybrid paper, and her own error rate as a consequence. So what is the big deal?

One of the reasons I want to ask this question is because it literally hurts to see how much of the wikipedia article on Statistical hypothesis testing is devoted to lambasting hybrid. Following Halpin & Stam, it claims that a a certain Lindquist is to blame (there is even a large scan of his textbook with "errors" highlighted in yellow), and of course the wiki article about Lindquist himself starts with the same accusation. But then, maybe I am missing something.

## Quotes

Gigerenzer: What has become institutionalized as inferential statistics in psychology is not Fisherian statistics. It is an incoherent mishmash of some of Fisher's ideas on one hand, and some of the ideas of Neyman and E. S. Pearson on the other. I refer to this blend as the "hybrid logic" of statistical inference.

Goodman: The [Neyman-Pearson] hypothesis test approach offered scientists a Faustian bargain -- a seemingly automatic way to limit the number of mistaken conclusions in the long run, but only by abandoning the ability to measure evidence [a la Fisher] and assess truth from a single experiment.

Hubbard & Bayarri: Classical statistical testing is an anonymous hybrid of the competing and frequently contradictory approaches [...]. In particular, there is a widespread failure to appreciate the incompatibility of Fisher's evidential $p$ value with the Type I error rate, $\alpha$, of Neyman-Pearson statistical orthodoxy. [...] As a prime example of the bewilderment arising from [this] mixing [...], consider the widely unappreciated fact that the former's $p$ value is incompatible with the Neyman-Pearson hypothesis test in which it has become embedded. [...] For example, Gibbons and Pratt [...] erroneously stated: "Reporting a P-value, whether exact or within an interval, in effect permits each individual to choose his own level of significance as the maximum tolerable probability of a Type I error."

Halpin & Stam: Lindquist's 1940 text was an original source of the hybridization of the Fisher and Neyman-Pearson approaches. [...] rather than adhering to any particular interpretation of statistical testing, psychologists have remained ambivalent about, and indeed largely unaware of, the conceptual difficulties implicated by the Fisher and Neyman-Pearson controversy.

Lew: What we have is a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence.

-
+1 for this well researched (even if long) question. It would help I think to perhaps continue to specify what exactly is confusing. Is it enough to know that for Fisher there doesnt exist an alternative hypothesis at all whereas for NP the world of possibilities is exhausted with both null and alternative? Seems incoherent enough to me but alas I do the hybrid thing all the time because you cant avoid, so ingrained has it become. – Momo Aug 21 '14 at 13:15
@Momo: to you question about "what exactly is confusing" -- well, confusing is the frenzy of the anti-hybrid rhetoric. "Incoherent mishmash" are strong words, so I would like to see a pretty bad inconsistency. What you said about alternative hypothesis does not sound as such to me (in the garden variety case of $H_0: \mu=0$ the alternative is obviously $H_1: \mu \ne 0$, and I don't see much room for inconsistency), but if I am missing your point then maybe you would like to provide it as an answer. – amoeba Aug 21 '14 at 15:42
Having just read Lew (and realizing I'd read it before, probably around 2006), I found it quite good, but I don't think it represents how I use p-values. My significance levels - on the rare occasions I use hypothesis testing at all* - are always up front, and where I have any control over sample size, after consideration of power, some consideration of the cost of the two error types and so on - essentially Neyman-Pearson. I still quote p-values, but not in the framework of Fisher's approach .... (ctd) – Glen_b Aug 22 '14 at 2:26
(ctd) ... * (I often steer people away from hypothesis testing - so often their actual questions are related to measuring effects, and are better answered by constructing intervals). The specific problem Lew raised for the 'hybrid' procedure applies to something I don't do and would tend to caution people against doing. If there are people really doing the mix of approaches he implies, the paper seems fine. The earlier discussion of the meaning of p-values and the history of the approaches seems excellent. – Glen_b Aug 22 '14 at 2:28
@Glen_b, Lew's historical overview is very nice and clear, I fully agree. My trouble is specifically with the hybrid issue (section "Which approach is most used?"). Certainly there are people doing what he describes there, i.e. reporting the strongest of p<.001, <.01, or <.05; I see it all the time in neuroscience. Consider one of the cases when you do use testing. You choose e.g. alpha=.05, and follow the NP framework. When you get p=.00011, is your certainty about H1 and your choice of wording going to be different from when you would get p=.049? If so, it is hybrid! If not, how come? – amoeba Aug 22 '14 at 11:02

I believe the papers, articles, posts e.t.c. that you diligently gathered, contain enough information and analysis as to where and why the two approaches differ. But being different does not mean being incompatible.

The problem with the "hybrid" is that it is a hybrid and not a synthesis, and this is why it is treated by many as a hybris, if you excuse the word-play.
Not being a synthesis, it does not attempt to combine the differences of the two approaches, and either create one unified and internally consistent approach, or keep both approaches in the scientific arsenal as complementary alternatives, in order to deal more effectively with the very complex world we try to analyze through Statistics (thankfully, this last thing is what appears to be happening with the other great civil war of the field, the frequentist-bayesian one).

The dissatisfaction with it I believe comes from the fact that it has indeed created misunderstandings in applying the statistical tools and interpreting the statistical results, mainly by scientists that are not statisticians, misunderstandings that can have possibly very serious and damaging effects (thinking about the field of medicine helps giving the issue its appropriate dramatic tone). This misapplication, is I believe, accepted widely as a fact-and in that sense, the "anti-hybrid" point of view can be considered as widespread (at least due to the consequences it had, if not for its methodological issues).

I see the evolution of the matter so far as a historical accident (but I don't have a $p$-value or a rejection region for my hypothesis), due to the unfortunate battle between the founders. Fisher and Neyman/Pearson have fought bitterly and publicly for decades over their approaches. This created the impression that here is a dichotomous matter: the one approach must be "right", and the other must be "wrong".

The hybrid emerged, I believe, out of the realization that no such easy answer existed, and that there were real-world phenomena to which the one approach is better suited than the other (see this post for such an example, according to me at least, where the Fisherian approach seems more suitable). But instead of keeping the two "separate and ready to act", they were rather superfluously patched together.

I offer a source which summarizes this "complementary alternative" approach: Spanos, A. (1999). Probability theory and statistical inference: econometric modeling with observational data. Cambridge University Press., ch. 14, especially Section 14.5, where after presenting formally and distinctly the two approaches, the author is in a position to point to their differences clearly, and also argue that they can be seen as complementary alternatives.

-
(+1) I appreciate your comments and agree with many of them. But I am not sure what exactly you are referring to when you say that the hybrid "created misunderstandings" (and moreover, that this is "accepted widely as a fact"). Could you give some examples? To be an attack on the hybrid, it should be examples of misunderstandings that do not arise in either F or N-P approaches alone. Are you referring to the potential confusion between $p$ and $\alpha$ that I mentioned in my question, or to something else? Apart from that, I am already reading Section 14.5 in Spanos, thanks. – amoeba Aug 21 '14 at 16:07
The obvious issue is indeed the $p-\alpha$ issue. More subtle and I believe more important, is the fact that the hybrid mixes the exploratory flavor of Fisher (which more over leaves the matter of decision to the researcher), with the more formal approach of N-P. So researchers approached the matter in a Fisherian spirit, but then claimed the strong "rejection/acceptance" weight of the N-P approach, which in principle gives more credibility to the conclusions. CONTD – Alecos Papadopoulos Aug 21 '14 at 16:18
CONTD For me, this is the "have your cake and eat it too" issue of the hybrid approach. For example, an N-P approach without power-test calculations should be unthinkable, but all the time we see test posed in the N-P framework, but no mention about power calculations. – Alecos Papadopoulos Aug 21 '14 at 16:19

My own take on my question is that there is nothing particularly incoherent in the hybrid (i.e. accepted) approach. But as I was not sure if I am maybe failing to comprehend the validity of the arguments presented in the anti-hybrid papers, I was happy to find the discussion published together with this paper:

Unfortunately, two replies published as a discussion were not formatted as separate articles and so cannot be properly cited. Still, I would like to quote from both of them:

Berk: The theme of Sections 2 and 3 seems to be that Fisher did not like what Neyman and Pearson did, and Neyman did not like what Fisher did, and therefore we should not do anything that combines the two approaches. There is no escaping the premise here, but the reasoning escapes me.

Carlton: the authors adamantly insist that most confusion stems from the marriage of Fisherian and Neyman-Pearsonian ideas, that such a marriage is a catastrophic error on the part of modern statisticians [...] [T]hey seem intent on establishing that P values and Type I errors cannot coexist in the same universe. It is unclear whether the authors have given any substantive reason why we cannot utter "p value" and "Type I error" in the same sentence. [...] The "fact" of their [F and NP] incompatibility comes as surprising news to me, as I'm sure it does to the thousands of qualified statisticians reading the article. The authors even seem to suggest that among the reasons statisticians should now divorce these two ideas is that Fisher and Neyman were not terribly fond of each other (or each other's philosophies on testing). I have always viewed our current practice, which integrates Fisher's and Neyman's philosophies and permits discussion of both P values and Type I errors -- though certainly not in parallel -- as one of our discipline's greater triumphs.

Both responses are very worth reading. There is also a rejoinder by the original authors, which does not sound convincing to me at all.

-
It is one thing to co-exist, it is another for the one to be considered as the other. But indeed, this strand of anti-hybrid approach is in the spirit of "there can be no synthesis whatsoever" -which I strongly disagree with. But I don't see the current hybrid as a successful marriage. – Alecos Papadopoulos Aug 21 '14 at 16:30
The problem with those quotes is that they do not justify the "hybrid". This is what many "anti-hybrid" people are looking for, and the lack of one being presented front and center is a glaring red flag. Anyone who investigates the history can see it clearly arose from some informal process. We cannot deduce from this that it is wrong, but what rationale is it based upon? When is it not applicable? What evidence is there to support its use with regards to different problems? How does it compare with other approaches (e.g., pure fisher, pure N-P, Bayesian, non-nil null hypothesis, etc)? – Livid Feb 14 '15 at 22:29
@Livid, thanks for your comments, this is interesting, but I would like to refrain from further discussion here. I would rather encourage you to post a new answer, if you wish. But if you decide to do so, try to focus on the main issue, which is: what is so bad about "hybrid", as compared to both Fisher and N-P alone. You seem to hate the whole approach of significance testing, "nil null hypothesis", etc., but this is not what this question is about! – amoeba Feb 14 '15 at 23:33
@Livid: Hmmm, can you actually clarify why you say that is is a distinguishing characteristic of the hybrid? What would the null be in pure Fisher or in pure NP? Say you have two groups and want to test for a significant difference ("nil null"). Can't one approach this situation with all three approaches: pure Fisher, pure NP, and hybrid? – amoeba Feb 14 '15 at 23:42
@Livid, I understand your arguments against the nil null, I just think that this issue is orthogonal to the issue of hybrid. I have to refresh the anti-hybrid papers in memory, but as far as I remember their critique of the hybrid is not at all centered on the nil null. Instead, it is about combining Fisher and NP. Again, if you disagree with this, please consider posting an answer; for the moment, let's leave it at that. – amoeba Feb 15 '15 at 0:06

I fear that a real response to this excellent question would require a full-length paper. However, here are a couple of points that are not present in either the question or the current answers.

1. The error rate 'belongs' to the procedure but the evidence 'belongs' to the experimental results. Thus it is possible with multi-stage procedures with sequential stopping rules to have a result with very strong evidence against the null hypothesis but a not significant hypothesis test result. That can be thought of as a strong incompatibility.

2. If you are interested in the incompatibilities, you should be interested in the underlying philosophies. The philosophical difficulty comes from a choice between compliance with the Likelihood Principle and compliance with the Repeated Sampling Principle. The LP says roughly that, given a statistical model, the evidence in a dataset relevant to the parameter of interest is completely contained in the relevant likelihood function. The RSP says that one should prefer tests that give error rates in the long run that equal their nominal values.

-
J. O. Berger and R.L Wolpert's monograph "The Likelihood Principle" (2nd ed. 1988), is a calm, balanced, and good exposition of point 2., in my opinion. – Alecos Papadopoulos Aug 21 '14 at 23:55
Berger and Wolpert is indeed a good exposition, and authoritative too. However, I prefer the more practically directed and less mathemtatical book "Likelihood" by AWF Edwards. Still in print, I think. books.google.com.au/books/about/Likelihood.html?id=LL08AAAAIAAJ – Michael Lew Aug 22 '14 at 1:27
@MichaelLew has explained that a valid use of p values is a summary of effect size. He has done a great thing by writing this paper: arxiv.org/abs/1311.0081 – Livid Feb 15 '15 at 2:25
@Livid The paper is v interesting, but for the new reader it's worth noting the following: the main idea, that p values 'index' (presumably: are in one to one relation with) likelihood functions, is generally understood to be false because there are cases where the same likelihood corresponds to different p-values depending on the sampling scheme. This issue is discussed a bit in the paper, but indexing is a very unusual position (which doesn't necessarily make it wrong, of course). – conjugateprior Jan 9 at 14:15

An often seen (and supposedly accepted) union (or better: "hybrid") between the two approaches is as follows:

1. Set a prespecified level $\alpha$ (0.05 say)
2. Then test your hypothesis, e.g. $H_o: \mu = 0$ vs. $H_1: \mu \ne 0$
3. State the p value and formulate your decision based on the level $\alpha$:

If the resulting p value is below $\alpha$, you could say

• "I reject $H_o$" or
• "I reject $H_o$" in favor of $H_1$" or
• "I am $100\% \cdot (1-\alpha)$ certain that $H_1$ holds"

If the p value is not small enough, you would say

• "I cannot reject $H_o$" or
• "I cannot reject $H_o$ in favor of $H_1$"

Here, aspects from Neyman-Pearson are:

• You decide something
• You have an alternative hypothesis at hand (although it is just the contrary of $H_o$)
• You know the type I error rate

Fisherian aspects are:

• You state the p value. Any reader has thus the possibility to use its own level (e.g. strictly correcting for multiple testing) for decision
• Basically, only the null hypothesis is required since the alternative is just the contrary
• You don't know the type II error rate. (But you could immediately get it for specific values of $\mu \ne 0$.)

While it is good to be aware of the discussion about the philosophical problems of Fisher's, NP's or this hybrid approach (as taught in almost religious frenzy by some), there are much more relevant issues in statistics to fight against:

• Asking uninformative questions (like binary yes/no questions instead of quantitative "how much" questions, i.e. using tests instead of confidence intervals)
• Data driven analysis methods that lead to biased results (stepwise regression, testing assumptions etc.)
• Choosing wrong tests or methods
• Misinterpreting results
• Using classic statistics for non-random samples
-
(+1) This is a good description of the hybrid (and why exactly it is hybrid), but you did not explicitly say what your evaluation of it is. Do you agree that what you described is an "incoherent mishmash"? If so, why? Or do you think it is a reasonable procedure? If so, do the people claiming it is incoherent have a point, or are they simply wrong? – amoeba Aug 21 '14 at 15:35
I often test hypotheses in exactly this manner... But there are other mish mashs I would not accept (e.g. not showing p values above $\alpha$) etc. – Michael M Aug 21 '14 at 15:50

accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

Short answer: the use of a nil (no difference, no correlation) null hypothesis irregardless of the context. Everything else is a "misuse" by people who have created myths for themselves about what the process can achieve. The myths arise from people attempting to reconcile their (sometimes appropriate) use of trust in authority and consensus heuristics with the inapplicability of the procedure to their problem.

As far as I know Gerd Gigerenzer came up with term "hybrid":

I asked the author [a distinguished statistical textbook author, whose book went through many editions, and whose name does not matter] why he removed the chapter on Bayes as well as the innocent sentence from all subsequent editions. “What made you present statistics as if it had only a single hammer, rather than a toolbox? Why did you mix Fisher’s and Neyman–Pearson’s theories into an inconsistent hybrid that every decent statistician would reject?”

To his credit, I should say that the author did not attempt to deny that he had produced the illusion that there is only one tool. But he let me know who was to blame for this. There were three culprits: his fellow researchers, the university administration, and his publisher. Most researchers, he argued, are not really interested in statistical thinking, but only in how to get their papers published [...]

The null ritual:

1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research hypothesis or of any alternative substantive hypotheses.

2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as $p < 0.05$, $p < 0.01$ , or $p < 0.001$ (whichever comes next to the obtained $p$-value).

3. Always perform this procedure.

Gigerenzer, G (November 2004). "Mindless statistics". The Journal of Socio-Economics 33 (5): 587–606. doi:10.1016/j.socec.2004.09.033.

Edit: And we should always need to mention, because the "hybrid" is so slippery and ill-defined, that using the nil null to get a p-value is perfectly fine as a way to compare effect sizes given different sample sizes. It is the "test" aspect that introduces the problem.

Edit 2: @amoeba A p-value can be fine as a summary statistic, in this case the nil null hypothesis is just an arbitrary landmark: http://arxiv.org/abs/1311.0081. However, as soon as you start trying to draw a conclusion or make a decision (ie "test" the null hypothesis) it stops making sense. In the comparing two groups example, we want to know how different two groups are and the various possible explanations there may be for differences of that magnitude and type.

The p value can be used as a summary statistic telling us the magnitude of the difference. However, using it to "disprove/reject" zero difference serves no purpose that I can tell. Also, I think many of these study designs that compare average measurements of living things at a single timepoint are misguided. We should want to observe how individual instances of the system change over time, then come up with a process that explains the pattern observed (including any group differences).

-
+1, Thanks for your answer and for the link. It seems I haven't read this particular paper, I will take a look. As I said before, I was under impression that "nil null" is an issue orthogonal to the issue of "hybrid", but I guess I should re-read Gigerenzer's writings to check that. Will try to find time in the following days. Apart from that: could you please clarify your last paragraph ("edit")? Did I understand correctly that you meant that having a nil null when comparing two effect sizes is okay, but having a nil null when comparing an effect size to zero is not okay? – amoeba Feb 15 '15 at 14:19