# What's the distinction between a “Null hypothesis statistical test” and any other test?

A recent hot topic of discussion concerns a journal banning the use of "null hypothesis statistical testing procedures (NHSTPs)" from articles submitted to the journal. I see this term used by some writers, but I don't understand what distinction they are trying to make. Is an NHSTP something different from a "test of hypothesis" or a "significance test"?

• An acronym that is somewhat more often used is NHST (without P in the end). Both acronyms seem to be pejorative terms used by writers who hate it (on in the context of such polemics). You can glance at the google scholar search for NHST+null (1670 results, vs only 145 for NHSTP+null) -- it's all about problems and troubles with it. Another related pejorative term is "null ritual". All of that means what you think it means, but pronounced with strong disgust! – amoeba Mar 4 '15 at 21:47
• There are tests that are notionally quite different from the more usual NHST, such as equivalence tests - though mechanically (but unsurprisingly) they make use of a closely related framework. However, I imagine the editor of the journal in question would probably argue against those, too. – Glen_b Mar 5 '15 at 2:49
• Related (but not a duplicate): What is the difference between “testing of hypothesis” and “test of significance”? Also, I should point out that the editors of this journal clearly do not try to make a distinction between any of that! They ban all kind of hypothesis/significance/whatever testing, be it Fisher, Neyman-Pearson, or hybrid. The distinction is only made in the context of Fisher vs. Neyman-Pearson (vs. hybrid) holy war, which is not what this particular journal ban was about. – amoeba Mar 5 '15 at 10:51
• So, @Livid, you are saying that "NHST" refers to the way statistical tests are usually used in practice, as distinguished from both the Fisher and N-P paradigms? I guess that, if in turn that implies routine and unthinking turning of the crank, then I'd agree it is a pejorative term. – Russ Lenth Mar 5 '15 at 18:10
• @rvl Here is the paper (+ commentary) I was thinking of earlier: Precis of Statistical significance: Rationale, validity, and utility. Siu L. Chow. BEHAVIORAL AND BRAIN SCIENCES (1998) 21, 169–239 – Livid Mar 6 '15 at 13:25

Background: The editorial in question is this one from Basic and Applied Social Psychology, a journal with a 2015 impact factor of 1.168, i.e., not highly quotable.

Re: OP question, i.e., Is an NHSTP something different from a "test of hypothesis" or a "significance test"? The applicable editorial statements are

1) "...the null hypothesis significance testing procedure (NHSTP)  is invalid..." [Sic, with alpha = 0.05]
2) "...authors will have to remove all vestiges of the NHSTP (p-values, t-values, F-values, statements about ‘‘significant’’ differences or lack thereof, and so on)."
3) "...confidence intervals [Sic, 95%] also are banned from BASP."
4) "...Bayesian procedures are neither required nor banned from BASP." [Sic, depends on which ones, they are either banned or not.]
5) "Are any inferential statistical procedures required?...No..."


The motivation offered for this is in part "...the $p<.05$ bar is too easy to pass and sometimes serves as an excuse for lower quality research. We hope and anticipate that banning the NHSTP will have the effect of increasing the quality of submitted manuscripts by liberating authors from the stultified structure of NHSTP thinking thereby eliminating an important obstacle to creative thinking."

Answer to OP: These editors would probably claim a test of significance is often an improper test of hypothesis. For example, they state that "...Bayesian proposals that at least somewhat circumvent the Laplacian assumption [Sic, I know nothing a priori]...[such that] there might even be cases where there are strong grounds for assuming that the numbers really are there..." This relates in part to the Fisher versus Neyman and Pearson argument as pointed out above by @Livid and for which the editorial would side with Fisher.

Discussion: I am a firm believer in intellectual humility as a fundamental, and indispensable tenet of scientific method. If I, as a researcher, am not allowed to proceed from an assumption-less initial premise in which all prior theory is disbelieved, then I will lose all of my ability to examine data with a creative and open mind. The premise that all numerical processing must be absolute truth is an exposition of cupidity that is sublime. The only truth is data, and I would humbly paraphrase Box by stating that all models are false, especially and most certainly those that presume that any truth arises from anything that is not identically the data itself. That does not mean that I have to choose between Fisher and Neyman/Pearson, rather that I firmly believe neither premise taken alone, but rather examine things exhaustively until my hypotheses are supported and/or rejected to self-consistency of the ensemble. Only self-consistency can be used as a criterion, as no analysis can reveal an absolute truth.

My way of doing things is not for everyone. Many prefer to plan testing in a rigid controlled experiment design that I would call 'top down'. However, controlled experiments are inefficient for data mining, pattern recognition, and generating hypotheses. They are useful for testing narrow questions, and that is when the controversy about NHSTP can arise. Without supporting evidence, e.g., an entire structure of self-consistency to rely on, any one test is open to criticism. This might be considered as Bonferroni in reverse; if multiple tests lead to an inescapably self-consistent ensemble, the chance of the ensemble occurring by chance alone is diminished. In planning experiments for psychology, the nonsense about not using $p<0.05$ is due to not also testing all of the implications of any particular test outcome, and if one cannot tolerate a type I error of $0.05$ because the experimental design is so rigid, restricted and narrow, then use $0.001$. However, to ban a particular statistical method because it is being used mindlessly and that mindless work passes muster when reviewed merely means that the editors are not identifying low quality work prior to agreeing to review it, and do not approach qualified reviewers. Certainly one cannot establish a reasonable conviction based on a single piece of circumstantial evidence. Rather, an ensemble of circumstantial evidence leads to a reasonable conviction. Eliminating an entire category of evidence because it is circumstantial will not improve a journal's content.

• "... as a fundamental, and indispensable tenant of scientific method ..." -- I expect you mean tenet rather than tenant. – Glen_b Nov 25 '16 at 4:41
• @Glen_b Je tiens à vous remercier. I should have known better, but expect it is all wrong anyway. Ergo, for the rest, any thoughts? – Carl Nov 25 '16 at 5:06