Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible" and hence the resulting "hybrid" is an "incoherent mishmash". I will provide a bibliography and some quotes below, but for now suffice it to say that there is a lot written about that in the wikipedia article on Statistical hypothesis testing. Here on CV, this point was repeatedly made by @Michael Lew.

1. Why are F and N-P approaches claimed to be incompatible and why is the hybrid claimed to be incoherent? Note that I read at least six anti-hybrid papers (see below), but still fail to understand the problem or the argument. Note also, that I am not suggesting to debate if F or N-P is a better approach; neither am I offering to discuss frequentist vs. Bayesian frameworks. Instead, the question is: accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

2. How accepted is this anti-hybrid point of view among professional statisticians? Is it rather a freak position, or maybe a trivial commonplace?

Here is how I understand the situation. Fisher's approach is to compute the $p$-value and take it as an evidence against the null hypothesis. The smaller the $p$, the more convincing the evidence. The researcher is supposed to combine this evidence with his background knowledge, decide if it is convincing enough, and proceed accordingly. (Note that Fisher's views changed over the years, but this is what he seems to have eventually converged to.) In contrast, Neyman-Pearson approach is to choose $\alpha$ ahead of time and then to check if $p\le\alpha$; if so, call it significant and reject the null hypothesis (here I omit large part of the N-P story that has no relevance for the current discussion). See also an excellent reply by @gung in When to use Fisher and Neyman-Pearson framework?

The hybrid approach is to compute the $p$-value, report it (implicitly assuming that the smaller the better), and also call the results significant if $p\le\alpha$ (usually $\alpha=0.05$) and nonsignificant otherwise. This is supposed to be incoherent. How can it be invalid to do two valid things simultaneously, beats me.

As particularly incoherent the anti-hybridists view the widespread practice of reporting $p$-values as $p<0.05$, $p<0.01$, or $p<0.001$ (or even $p\ll0.0001$), where always the strongest inequality is chosen. The argument seems to be that (a) the strength of evidence cannot be properly assessed as exact $p$ is not reported, and (b) people tend to interpret the right-hand number in the inequality as $\alpha$ and view it as type I error rate, and that is wrong. I fail to see a big problem here. First, reporting exact $p$ is certainly a better practice, but nobody really cares if $p$ is e.g. $0.02$ or $0.03$, so rounding it on a log scale is not soooo bad (and going below $\sim 0.0001$ does not make sense anyway, see How should tiny p-values be reported?). Second, if the consensus is to call everything below $0.05$ significant, then error rate will be $\alpha=0.05$ and $p \ne \alpha$, as @gung explains in Interpretation of p-value in hypothesis testing. Even though this is potentially a confusing issue, it does not strike me as more confusing than other issues in statistical testing (outside of the hybrid). Also, every reader can have her own favourite $\alpha$ in mind when reading a hybrid paper, and her own error rate as a consequence. So what is the big deal?

One of the reasons I want to ask this question is because it literally hurts to see how much of the wikipedia article on Statistical hypothesis testing is devoted to lambasting hybrid. Following Halpin & Stam, it claims that a a certain Lindquist is to blame (there is even a large scan of his textbook with "errors" highlighted in yellow), and of course the wiki article about Lindquist himself starts with the same accusation. But then, maybe I am missing something.

Quotes

Gigerenzer: What has become institutionalized as inferential statistics in psychology is not Fisherian statistics. It is an incoherent mishmash of some of Fisher's ideas on one hand, and some of the ideas of Neyman and E. S. Pearson on the other. I refer to this blend as the "hybrid logic" of statistical inference.

Goodman: The [Neyman-Pearson] hypothesis test approach offered scientists a Faustian bargain -- a seemingly automatic way to limit the number of mistaken conclusions in the long run, but only by abandoning the ability to measure evidence [a la Fisher] and assess truth from a single experiment.

Hubbard & Bayarri: Classical statistical testing is an anonymous hybrid of the competing and frequently contradictory approaches [...]. In particular, there is a widespread failure to appreciate the incompatibility of Fisher's evidential $p$ value with the Type I error rate, $\alpha$, of Neyman-Pearson statistical orthodoxy. [...] As a prime example of the bewilderment arising from [this] mixing [...], consider the widely unappreciated fact that the former's $p$ value is incompatible with the Neyman-Pearson hypothesis test in which it has become embedded. [...] For example, Gibbons and Pratt [...] erroneously stated: "Reporting a P-value, whether exact or within an interval, in effect permits each individual to choose his own level of significance as the maximum tolerable probability of a Type I error."

Halpin & Stam: Lindquist's 1940 text was an original source of the hybridization of the Fisher and Neyman-Pearson approaches. [...] rather than adhering to any particular interpretation of statistical testing, psychologists have remained ambivalent about, and indeed largely unaware of, the conceptual difficulties implicated by the Fisher and Neyman-Pearson controversy.

Lew: What we have is a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence.

-
+1 for this well researched (even if long) question. It would help I think to perhaps continue to specify what exactly is confusing. Is it enough to know that for Fisher there doesnt exist an alternative hypothesis at all whereas for NP the world of possibilities is exhausted with both null and alternative? Seems incoherent enough to me but alas I do the hybrid thing all the time because you cant avoid, so ingrained has it become. –  Momo Aug 21 at 13:15
And a note on the wording in the anti-hybrid papers (or other contested things in ststs for that matter), never forget that they are often polemic to bring their point across. Rarely do these papers really reflect the subtelties, normativities and necessities involved in such a debate. Everyones a critic. –  Momo Aug 21 at 13:22
@Momo: to you question about "what exactly is confusing" -- well, confusing is the frenzy of the anti-hybrid rhetoric. "Incoherent mishmash" are strong words, so I would like to see a pretty bad inconsistency. What you said about alternative hypothesis does not sound as such to me (in the garden variety case of $H_0: \mu=0$ the alternative is obviously $H_1: \mu \ne 0$, and I don't see much room for inconsistency), but if I am missing your point then maybe you would like to provide it as an answer. –  amoeba Aug 21 at 15:42
Having just read Lew (and realizing I'd read it before, probably around 2006), I found it quite good, but I don't think it represents how I use p-values. My significance levels - on the rare occasions I use hypothesis testing at all* - are always up front, and where I have any control over sample size, after consideration of power, some consideration of the cost of the two error types and so on - essentially Neyman-Pearson. I still quote p-values, but not in the framework of Fisher's approach .... (ctd) –  Glen_b Aug 22 at 2:26
(ctd) ... * (I often steer people away from hypothesis testing - so often their actual questions are related to measuring effects, and are better answered by constructing intervals). The specific problem Lew raised for the 'hybrid' procedure applies to something I don't do and would tend to caution people against doing. If there are people really doing the mix of approaches he implies, the paper seems fine. The earlier discussion of the meaning of p-values and the history of the approaches seems excellent. –  Glen_b Aug 22 at 2:28

I believe the papers, articles, posts e.t.c. that you diligently gathered, contain enough information and analysis as to where and why the two approaches differ. But being different does not mean being incompatible.

The problem with the "hybrid" is that it is a hybrid and not a synthesis, and this is why it is treated by many as a hybris, if you excuse the word-play.
Not being a synthesis, it does not attempt to combine the differences of the two approaches, and either create one unified and internally consistent approach, or keep both approaches in the scientific arsenal as complementary alternatives, in order to deal more effectively with the very complex world we try to analyze through Statistics (thankfully, this last thing is what appears to be happening with the other great civil war of the field, the frequentist-bayesian one).

The dissatisfaction with it I believe comes from the fact that it has indeed created misunderstandings in applying the statistical tools and interpreting the statistical results, mainly by scientists that are not statististicians, misunderstandings that can have possibly very serious and damaging effects (thinking about the field of medicine helps giving the issue its appropriate dramatic tone). This misapplication, is I believe, accepted widely as a fact-and in that sense, the "anti-hybrid" point of view can be considered as widespread (at least due to the consequences it had, if not for its methodological issues).

I see the evolution of the matter so far as a historical accident (but I don't have a $p$-value or a rejection region for my hypothesis), due to the unfortunate battle between the founders. Fisher and Neyman/Pearson have fought bitterly and publicly for decades over their approaches. This created the impression that here is a dichotomous matter: the one approach must be "right", and the other must be "wrong".

The hybrid emerged, I believe, out of the realization that no such easy answer existed, and that there were real-world phenomena to which the one approach is better suited than the other (see this post for such an example, according to me at least, where the Fisherian approach seems more suitable). But instead of keeping the two "separate and ready to act", they were rather superfluously patched together.

I offer a source which summarizes this "complementary alternative" approach: Spanos, A. (1999). Probability theory and statistical inference: econometric modeling with observational data. Cambridge University Press., ch. 14, especially Section 14.5, where after presenting formally and distinctly the two approaches, the author is in a position to point to their differences clearly, and also argue that they can be seen as complementary alternatives.

-
(+1) I appreciate your comments and agree with many of them. But I am not sure what exactly you are referring to when you say that the hybrid "created misunderstandings" (and moreover, that this is "accepted widely as a fact"). Could you give some examples? To be an attack on the hybrid, it should be examples of misunderstandings that do not arise in either F or N-P approaches alone. Are you referring to the potential confusion between $p$ and $\alpha$ that I mentioned in my question, or to something else? Apart from that, I am already reading Section 14.5 in Spanos, thanks. –  amoeba Aug 21 at 16:07
The obvious issue is indeed the $p-\alpha$ issue. More subtle and I believe more important, is the fact that the hybrid mixes the exploratory flavor of Fisher (which more over leaves the matter of decision to the researcher), with the more formal approach of N-P. So researchers approached the matter in a Fisherian spirit, but then claimed the strong "rejection/acceptance" weight of the N-P approach, which in principle gives more credibility to the conclusions. CONTD –  Alecos Papadopoulos Aug 21 at 16:18
CONTD For me, this is the "have your cake and eat it too" issue of the hybrid approach. For example, an N-P approach without power-test calculations should be unthinkable, but all the time we see test posed in the N-P framework, but no mention about power calculations. –  Alecos Papadopoulos Aug 21 at 16:19

My own take on my question is that there is nothing particularly incoherent in the hybrid (i.e. accepted) approach. But as I was not sure if I am maybe failing to comprehend the validity of the arguments presented in the anti-hybrid papers, I was happy to find the discussion published together with this paper:

Unfortunately, two replies published as a discussion were not formatted as separate articles and so cannot be properly cited. Still, I would like to quote from both of them:

Berk: The theme of Sections 2 and 3 seems to be that Fisher did not like what Neyman and Pearson did, and Neyman did not like what Fisher did, and therefore we should not do anything that combines the two approaches. There is no escaping the premise here, but the reasoning escapes me.

Carlton: the authors adamantly insist that most confusion stems from the marriage of Fisherian and Neyman-Pearsonian ideas, that such a marriage is a catastrophic error on the part of modern statisticians [...] [T]hey seem intent on establishing that P values and Type I errors cannot coexist in the same universe. It is unclear whether the authors have given any substantive reason why we cannot utter "p value" and "Type I error" in the same sentence. [...] The "fact" of their [F and NP] incompatibility comes as surprising news to me, as I'm sure it does to the thousands of qualified statisticians reading the article. The authors even seem to suggest that among the reasons statisticians should now divorce these two ideas is that Fisher and Neyman were not terribly fond of each other (or each other's philosophies on testing). I have always viewed our current practice, which integrates Fisher's and Neyman's philosophies and permits discussion of both P values and Type I errors -- though certainly not in parallel -- as one of our discipline's greater triumphs.

Both responses are very worth reading. There is also a rejoinder by the original authors, which does not sound convincing to me at all.

-
It is one thing to co-exist, it is another for the one to be considered as the other. But indeed, this strand of anti-hybrid approach is in the spirit of "there can be no synthesis whatsoever" -which I strongly disagree with. But I don't see the current hybrid as a successful marriage. –  Alecos Papadopoulos Aug 21 at 16:30

An often seen (and supposedly accepted) union (or better: "hybrid") between the two approaches is as follows:

1. Set a prespecified level $\alpha$ (0.05 say)
2. Then test your hypothesis, e.g. $H_o: \mu = 0$ vs. $H_1: \mu \ne 0$
3. State the p value and formulate your decision based on the level $\alpha$:

If the resulting p value is below $\alpha$, you could say

• "I reject $H_o$" or
• "I reject $H_o$" in favor of $H_1$" or
• "I am $100\% \cdot (1-\alpha)$ certain that $H_1$ holds"

If the p value is not small enough, you would say

• "I cannot reject $H_o$" or
• "I cannot reject $H_o$ in favor of $H_1$"

Here, aspects from Neyman-Pearson are:

• You decide something
• You have an alternative hypothesis at hand (although it is just the contrary of $H_o$)
• You know the type I error rate

Fisherian aspects are:

• You state the p value. Any reader has thus the possibility to use its own level (e.g. strictly correcting for multiple testing) for decision
• Basically, only the null hypothesis is required since the alternative is just the contrary
• You don't know the type II error rate. (But you could immediately get it for specific values of $\mu \ne 0$.)

While it is good to be aware of the discussion about the philosophical problems of Fisher's, NP's or this hybrid approach (as taught in almost religious frenzy by some), there are much more relevant issues in statistics to fight against:

• Asking uninformative questions (like binary yes/no questions instead of quantitative "how much" questions, i.e. using tests instead of confidence intervals)
• Data driven analysis methods that lead to biased results (stepwise regression, testing assumptions etc.)
• Choosing wrong tests or methods
• Misinterpreting results
• Using classic statistics for non-random samples
-
(+1) This is a good description of the hybrid (and why exactly it is hybrid), but you did not explicitly say what your evaluation of it is. Do you agree that what you described is an "incoherent mishmash"? If so, why? Or do you think it is a reasonable procedure? If so, do the people claiming it is incoherent have a point, or are they simply wrong? –  amoeba Aug 21 at 15:35
I often test hypotheses in exactly this manner... But there are other mish mashs I would not accept (e.g. not showing p values above $\alpha$) etc. –  Michael Mayer Aug 21 at 15:50

I fear that a real response to this excellent question would require a full-length paper. However, here are a couple of points that are not present in either the question or the current answers.

1. The error rate 'belongs' to the procedure but the evidence 'belongs' to the experimental results. Thus it is possible with multi-stage procedures with sequential stopping rules to have a result with very strong evidence against the null hypothesis but a not significant hypothesis test result. That can be thought of as a strong incompatibility.

2. If you are interested in the incompatibilities, you should be interested in the underlying philosophies. The philosophical difficulty comes from a choice between compliance with the Likelihood Principle and compliance with the Repeated Sampling Principle. The LP says roughly that, given a statistical model, the evidence in a dataset relevant to the parameter of interest is completely contained in the relevant likelihood function. The RSP says that one should prefer tests that give error rates in the long run that equal their nominal values.

-
J. O. Berger and R.L Wolpert's monograph "The Likelihood Principle" (2nd ed. 1988), is a calm, balanced, and good exposition of point 2., in my opinion. –  Alecos Papadopoulos Aug 21 at 23:55
Berger and Wolpert is indeed a good exposition, and authoritative too. However, I prefer the more practically directed and less mathemtatical book "Likelihood" by AWF Edwards. Still in print, I think. books.google.com.au/books/about/Likelihood.html?id=LL08AAAAIAAJ –  Michael Lew Aug 22 at 1:27