There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible" and hence the resulting "hybrid" is an "incoherent mishmash". I will provide a bibliography and some quotes below, but for now suffice it to say that there is a lot written about that in the wikipedia article on Statistical hypothesis testing. Here on CV, this point was repeatedly made by @Michael Lew (see here and here).
My question is: why are F and N-P approaches claimed to be incompatible and why is the hybrid claimed to be incoherent? Note that I read at least six anti-hybrid papers (see below), but still fail to understand the problem or the argument. Note also, that I am not suggesting to debate if F or N-P is a better approach; neither am I offering to discuss frequentist vs. Bayesian frameworks. Instead, the question is: accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?
Here is how I understand the situation. Fisher's approach is to compute the $p$-value and take it as an evidence against the null hypothesis. The smaller the $p$, the more convincing the evidence. The researcher is supposed to combine this evidence with his background knowledge, decide if it is convincing enough, and proceed accordingly. (Note that Fisher's views changed over the years, but this is what he seems to have eventually converged to.) In contrast, Neyman-Pearson approach is to choose $\alpha$ ahead of time and then to check if $p\le\alpha$; if so, call it significant and reject the null hypothesis (here I omit large part of the N-P story that has no relevance for the current discussion). See also an excellent reply by @gung in When to use Fisher and Neyman-Pearson framework?
The hybrid approach is to compute the $p$-value, report it (implicitly assuming that the smaller the better), and also call the results significant if $p\le\alpha$ (usually $\alpha=0.05$) and nonsignificant otherwise. This is supposed to be incoherent. How can it be invalid to do two valid things simultaneously, beats me.
As particularly incoherent the anti-hybridists view the widespread practice of reporting $p$-values as $p<0.05$, $p<0.01$, or $p<0.001$ (or even $p\ll0.0001$), where always the strongest inequality is chosen. The argument seems to be that (a) the strength of evidence cannot be properly assessed as exact $p$ is not reported, and (b) people tend to interpret the right-hand number in the inequality as $\alpha$ and view it as type I error rate, and that is wrong. I fail to see a big problem here. First, reporting exact $p$ is certainly a better practice, but nobody really cares if $p$ is e.g. $0.02$ or $0.03$, so rounding it on a log scale is not soooo bad (and going below $\sim 0.0001$ does not make sense anyway, see How should tiny p-values be reported?). Second, if the consensus is to call everything below $0.05$ significant, then error rate will be $\alpha=0.05$ and $p \ne \alpha$, as @gung explains in Interpretation of p-value in hypothesis testing. Even though this is potentially a confusing issue, it does not strike me as more confusing than other issues in statistical testing (outside of the hybrid). Also, every reader can have her own favourite $\alpha$ in mind when reading a hybrid paper, and her own error rate as a consequence. So what is the big deal?
One of the reasons I want to ask this question is because it literally hurts to see how much of the wikipedia article on Statistical hypothesis testing is devoted to lambasting hybrid. Following Halpin & Stam, it claims that a a certain Lindquist is to blame (there is even a large scan of his textbook with "errors" highlighted in yellow), and of course the wiki article about Lindquist himself starts with the same accusation. But then, maybe I am missing something.
Gigerenzer, 1993, The superego, the ego, and the id in statistical reasoning -- introduced the term "hybrid" and called it "incoherent mishmash"
- See also more recent expositions by Gigerenzer et al.: e.g. Mindless statistics (2004) and The Null Ritual. What You Always Wanted to Know About Significance Testing but Were Afraid to Ask (2004).
Cohen, 1994, The Earth Is Round ($p<.05$) -- a very popular paper with almost 3k citations, mostly about different issues but favourably citing Gigerenzer
Hubbard & Bayarri, 2003, Confusion over measures of evidence ($p$'s) versus errors ($\alpha$'s) in classical statistical testing -- one of the more eloquent papers arguing against "hybrid"
Halpin & Stam, 2006, Inductive Inference or Inductive Behavior: Fisher and Neyman-Pearson Approaches to Statistical Testing in Psychological Research (1940-1960) [free after registration] -- blames Lindquist's 1940 textbook for introducing the "hybrid" approach
@Michael Lew, 2006, Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P -- a nice review and overview
Gigerenzer: What has become institutionalized as inferential statistics in psychology is not Fisherian statistics. It is an incoherent mishmash of some of Fisher's ideas on one hand, and some of the ideas of Neyman and E. S. Pearson on the other. I refer to this blend as the "hybrid logic" of statistical inference.
Goodman: The [Neyman-Pearson] hypothesis test approach offered scientists a Faustian bargain -- a seemingly automatic way to limit the number of mistaken conclusions in the long run, but only by abandoning the ability to measure evidence [a la Fisher] and assess truth from a single experiment.
Hubbard & Bayarri: Classical statistical testing is an anonymous hybrid of the competing and frequently contradictory approaches [...]. In particular, there is a widespread failure to appreciate the incompatibility of Fisher's evidential $p$ value with the Type I error rate, $\alpha$, of Neyman-Pearson statistical orthodoxy. [...] As a prime example of the bewilderment arising from [this] mixing [...], consider the widely unappreciated fact that the former's $p$ value is incompatible with the Neyman-Pearson hypothesis test in which it has become embedded. [...] For example, Gibbons and Pratt [...] erroneously stated: "Reporting a P-value, whether exact or within an interval, in effect permits each individual to choose his own level of significance as the maximum tolerable probability of a Type I error."
Halpin & Stam: Lindquist's 1940 text was an original source of the hybridization of the Fisher and Neyman-Pearson approaches. [...] rather than adhering to any particular interpretation of statistical testing, psychologists have remained ambivalent about, and indeed largely unaware of, the conceptual difficulties implicated by the Fisher and Neyman-Pearson controversy.
Lew: What we have is a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence.