73
$\begingroup$

On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers.

Specifically, they say (formatting and emphasis are mine):

  • [...] prior to publication, authors will have to remove all vestiges of the NHSTP [null hypothesis significance testing procedure] ($p$-values, $t$-values, $F$-values, statements about ‘‘significant’’ differences or lack thereof, and so on).

  • Analogous to how the NHSTP fails to provide the probability of the null hypothesis, which is needed to provide a strong case for rejecting it, confidence intervals do not provide a strong case for concluding that the population parameter of interest is likely to be within the stated interval. Therefore, confidence intervals also are banned from BASP.

  • [...] with respect to Bayesian procedures, we reserve the right to make case-by-case judgments, and thus Bayesian procedures are neither required nor banned from BASP.

  • [...] Are any inferential statistical procedures required? -- No [...] However, BASP will require strong descriptive statistics, including effect sizes.

Let us not discuss problems with and misuse of $p$-values here; there already are plenty of excellent discussions on CV that can be found by browsing the p-value tag. The critique of $p$-values often goes together with an advice to report confidence intervals for parameters of interest. For example, in this very well-argued answer @gung suggests to report effect sizes with confidence intervals around them. But this journal bans confidence intervals as well.

What are the advantages and disadvantages of such an approach to presenting data and experimental results as opposed to the "traditional" approach with $p$-values, confidence intervals, and significant/insignificant dichotomy? The reaction to this ban seems to be mostly negative; so what are the disadvantages then? American Statistical Association has even posted a brief discouraging comment on this ban, saying that "this policy may have its own negative consequences". What could these negative consequences be?

Or as @whuber suggested to put it, should this approach be advocated generally as a paradigm of quantitative research? And if not, why not?

PS. Note that my question is not about the ban itself; it is about the suggested approach. I am not asking about frequentist vs. Bayesian inference either. The Editorial is pretty negative about Bayesian methods too; so it is essentially about using statistics vs. not using statistics at all.

Other discussions: reddit, Gelman.

|cite|improve this question|||||
$\endgroup$
  • 14
    $\begingroup$ There is a one-to-one mapping between p-values and confidence intervals in linear regression models, so I don't see a strong reason why banning p-values but keeping confidence intervals would make much sense. But banning both p-values and confidence intervals leaves a gap in description of results... I wonder if they allow reporting standard errors (that would be another measure of the same one-to-one mapping group). $\endgroup$ – Richard Hardy Feb 25 '15 at 19:07
  • 7
    $\begingroup$ Everything could be misused so banning stuff on this condition is, well... strange. I am not the fan of p-values but this seems as a pretty naive approach to the problem. One thing is encouraging to use proper stuff, but banning things does not sound like a proper way to deal with the problem... $\endgroup$ – Tim♦ Feb 25 '15 at 21:23
  • 12
    $\begingroup$ Great idea. Using statistics just hides the unscientific nature of this field. $\endgroup$ – Aksakal Feb 25 '15 at 21:35
  • 4
    $\begingroup$ This seems like a complete overreaction to the frustration over the misuse of p values. I would be much happier with a ban on the misuse of p values rather than P values in general. $\endgroup$ – TrynnaDoStat Feb 26 '15 at 1:17
  • 8
    $\begingroup$ The 4th item in your list suggests they're not requiring point estimates, which would be inference, but effect sizes reported merely as descriptive statistics. (Nevertheless, a few lines down in the editorial, "we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem". I look forward to the 2016 editorial's calling for research into formalizing this notion of stability & accounting quantitatively for the effects of sampling error.) $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 26 '15 at 18:55
23
$\begingroup$

The first sentence of the current 2015 editorial to which the OP links, reads:

The Basic and Applied Social Psychology (BASP) 2014 Editorial *emphasized* that the null hypothesis significance testing procedure (NHSTP) is invalid...

(my emphasis)

In other words, for the editors it is an already proven scientific fact that "null hypothesis significance testing" is invalid, and the 2014 editorial only emphasized so, while the current 2015 editorial just implements this fact.

The misuse (even maliciously so) of NHSTP is indeed well discussed and documented. And it is not unheard of in human history that "things get banned" because it has been found that after all said and done, they were misused more than put to good use (but shouldn't we statistically test that?). It can be a "second-best" solution, to cut what on average (inferential statistics) has come to losses, rather than gains, and so we predict (inferential statistics) that it will be detrimental also in the future.

But the zeal revealed behind the wording of the above first sentence, makes this look -exactly, as a zealot approach rather than a cool-headed decision to cut the hand that tends to steal rather than offer. If one reads the one-year older editorial mentioned in the above quote (DOI:10.1080/01973533.2014.865505), one will see that this is only part of a re-hauling of the Journal's policies by a new Editor.

Scrolling down the editorial, they write

...On the contrary, we believe that the p<.05 bar is too easy to pass and sometimes serves as an excuse for lower quality research.

So it appears that their conclusion related to their discipline is that null-hypotheses are rejected "too-often", and so alleged findings may acquire spurious statistical significance. This is not the same argument as the "invalid" dictum in the first sentence.

So, to answer to the question, it is obvious that for the editors of the journal, their decision is not only wise but already late in being implemented: they appear to think that they cut out what part of statistics has become harmful, keeping the beneficial parts -they don't seem to believe that there is anything here that needs replacing with something "equivalent".

Epistemologically, this is an instance where scholars of a social science partially retract back from an attempt to make their discipline more objective in its methods and results by using quantitative methods, because they have arrived at the conclusion (how?) that, in the end, the attempt created "more bad than good". I would say that this is a very important matter, in principle possible to have happened, and one that would require years of work to demonstrate it "beyond reasonable doubt" and really help your discipline. But just one or two editorials and papers published will most probably (inferential statistics) just ignite a civil war.

The final sentence of the 2015 editorial reads:

We hope and anticipate that banning the NHSTP will have the effect of increasing the quality of submitted manuscripts by liberating authors from the stultified structure of NHSTP thinking thereby eliminating an important obstacle to creative thinking. The NHSTP has dominated psychology for decades; we hope that by instituting the first NHSTP ban, we demonstrate that psychology does not need the crutch of the NHSTP, and that other journals follow suit.

|cite|improve this answer|||||
$\endgroup$
  • 5
    $\begingroup$ Yes...we have to be careful when writing tongue-in-cheek or sardonic replies on this site: they might be (completely) misunderstood! $\endgroup$ – whuber♦ Feb 25 '15 at 21:53
  • 4
    $\begingroup$ @naught101 ...that wouldn't be very diplomatic. Notice that the way the NHSTP is condemned, it spares the psychologists themselves that they have used it in all these decades. If it was written the way you propose, it would look much more like a direct attack on their colleagues as scientists. As it now stands essentially the text implies that psychologists full of good intentions have been unfortunately misled in using the approach, by "someone", which misused his "power of scientific authority" in the matter... Perhaps by evil statisticians driven by scientific imperialism? $\endgroup$ – Alecos Papadopoulos Feb 26 '15 at 1:09
  • 4
    $\begingroup$ A bad workman blames his tools. $\endgroup$ – naught101 Feb 26 '15 at 1:39
  • 3
    $\begingroup$ @BrianDHall I would suggest to look up more authoritative resources on the issues surrounding NHSTP (this site included), rather than the specific author's works on the issue. The matter is difficult and subtle -already from your comment one should discuss first the semantics around "accept" and "assert"... $\endgroup$ – Alecos Papadopoulos Feb 26 '15 at 3:51
  • 6
    $\begingroup$ @naught101: If you notice that the workman can't handle the chainsaw properly, you might not blame the tool. But you would still take it away from the workman, to prevent further harm ;-) $\endgroup$ – nikie Feb 26 '15 at 11:33
19
$\begingroup$

I feel that banning hypothesis tests is a great idea except for a select few "existence" hypotheses, e.g. testing the null hypothesis that there is not extra-sensory perception where all one would need to demonstrate to have evidence that ESP exists is non-randomness. But I think the journal missed the point that the main driver of poor research in psychology is the use of a threshold on $P$-values. It has been demonstrated in psychology and most other fields that a good deal of gaming goes on to arrive at $P < 0.05$. This includes hypothesis substitution, removing of observations, and subsetting data. It is thresholds that should be banned first.

The banning of confidence intervals is also overboard, but not for the reasons others have stated. Confidence intervals are useful only if one misinterprets them as Bayesian credible intervals (for suitable non-information priors). But they are still useful. The fact that their exact frequentist interpretation leads to nothing but confusion implies that we need to "get out of Dodge" and go Bayesian or likelihood school. But useful results can be obtained by misinterpreting good old confidence limits.

It is a shame that the editors of the journal misunderstood Bayesian statistics and don't know of the existence of pure likelihood inference. What they are seeking can be easily provided by Bayesian posterior distributions using slightly skeptical priors.

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ +1, thanks. Let me clarify regarding confidence intervals. Confidence intervals are related to standard errors, so the suggestion is probably to stop using those as well. Let's consider the simplest case: some value is measured across a group of $n$ subjects/objects; let's say the mean is 3. As far as I understand this journal suggests to report it simply as 3. But would you not want to see standard error as well, e.g. $3 \pm 0.5$? This of course means that 95% confidence interval is $3 \pm 1$, which also means that $p<0.05$, so it's all related. I am not sure how you suggest to report it. $\endgroup$ – amoeba says Reinstate Monica Feb 27 '15 at 12:00
  • 4
    $\begingroup$ I think of standard errors are oversimplified (because they assume symmetric distributions) but useful measures of precision, like mean squared error. You can think of a precision interval based on root mean squared error without envisioning probability coverage. So I don't see where any of this discussion implies de-emphasis of standard errors. And I wasn't suggesting that we stop using CLs. But the difficulty with CLs comes mainly from attempts at probability interpretations. $\endgroup$ – Frank Harrell Feb 27 '15 at 12:48
  • $\begingroup$ Hmmm. Interesting. To me it seems like there is such a small step from standard error to CI (a constant factor!), that treating them differently would be weird. But perhaps it is a semantic point; I guess what you mean is that people think about standard errors and CIs differently and tend to get more confused about CIs. I wonder what this particular journal policy says about standard errors (the Editorial doesn't mention them explicitly). $\endgroup$ – amoeba says Reinstate Monica Feb 27 '15 at 22:26
  • 2
    $\begingroup$ In symmetric situations, the standard error is a building block for a confidence interval. But in many cases the correct confidence interval is asymmetric so can't be based on a standard error at all. Some varieties of the bootstrap and back-transforming are two approaches of this type. Profile likelihood confidence intervals especially come to mind here. $\endgroup$ – Frank Harrell Feb 28 '15 at 12:11
  • $\begingroup$ @Frank Harrell - As for "pure likelihood inference" I agree that an emphasis toward summarization of the data's likelihood without embellishing it with thresholds appears to be the answer the editors were grasping for. A. W. F. Edwards' book "Likelihood" (1972) speaks directly to the editor's concern: "We may defer consideration of these arguments (e.g. significance testing) until later chapters, and pass immediately to the description of a procedure, based on Fisher's concept of Likelihood, which is open to none of these objects which may be levelled at significance tests." $\endgroup$ – John Mark Apr 20 '15 at 15:43
13
$\begingroup$

I see this approach as an attempt to address the inability of social psychology to replicate many previously published 'significant findings.'

Its disadvantages are:

  1. that it doesn't address many of the factors leading to spurious effects. E.g.,

    • A) People can still peek at their data and stop running their studies when an effect size strikes them as being sufficiently large to be of interest.

    • B) Large effects sizes will still appear to have large power in retrospective assessments of power.

    • C) People will still fish for interesting and big effects (testing a bunch of hypotheses in an experiment and then reporting the one that popped up) or

    • D) pretend that an unexpected weird effect was expected all along.

    Shouldn't efforts be made to address these issues first?

  2. As a field going forwards it will make a review of past findings pretty awful. There is no way to quantitatively assess the believability of different studies. If every journal implemented this approach, you'll have a bunch of social scientists saying there is evidence for X when it is totally unclear how believable X is and scientists arguing about how to interpret a published effect or arguing about whether it is important or worth talking about. Isn't this the point of having stats? To provide a consistent way to assess numbers. In my opinion, this new approach would cause a mess if it was widely implemented.

  3. This change does not encourage researchers to submit the results of studies with small effect sizes so it doesn't really address the file-drawer effect (or are they going to publish findings with large n's regardless of effect size?). If we published all results of carefully designed studies, then even though the believability of results of individual studies may be uncertain, meta-analyses and reviews of studies that supplied statistical analysis would do a much better job at identifying the truth.

|cite|improve this answer|||||
$\endgroup$
  • 2
    $\begingroup$ @captain_ahab Regarding point 3, we must mention that the previous editorial (2014) of the Editor explicitly encouraged the submission of "null-effect" studies. $\endgroup$ – Alecos Papadopoulos Feb 27 '15 at 11:48
  • 1
    $\begingroup$ I can't seem to find a comment in the editorial discussing any criteria for publication except for the need have larger sample sizes than normal (how they are planning on identifying acceptable n's without inferential statistics is unclear to me). To me there is no emphasis in this editorial that they don't care what the effect size is. It seems to me that they will still be looking for interesting effects and interesting stories, which I think is the bigger problem in social science work (i.e., the post-hoc search for interesting effects and stories). $\endgroup$ – captain_ahab Feb 27 '15 at 19:34
  • 2
    $\begingroup$ What seems like a better solution is that all scientists must log the hypothesis, basic rational, power of and analytics approach of a study in a PUBLIC place BEFORE running the study. And then being limited to publishing that study in the prescribe manner. If an unexpected interesting effect is found, they should publicly log then run a new study that examines that effect. This approach while controlling for false positives would also enable scientists to demonstrate their productivity without publishing new effects. $\endgroup$ – captain_ahab Feb 27 '15 at 19:39
7
$\begingroup$

I came across a wonderful quote that almost argues for the same point, but not quite -- since it is an opening paragraph in a textbook that is mostly about frequentist statistics and hypothesis testing.

It is widely held by non-statisticians, like the author, that if you do good experiments statistics are not necessary. They are quite right. [...] The snag, of course, is that doing good experiments is difficult. Most people need all the help they can get to prevent them making fools of themselves by claiming that their favourite theory is substantiated by observations that do nothing of the sort. And the main function of that section of statistics that deals with tests of significance is to prevent people making fools of themselves. From this point of view, the function of significance tests is to prevent people publishing experiments, not to encourage them. Ideally, indeed, significance tests should never appear in print, having been used, if at all, in the preliminary stages to detect inadequate experiments, so that the final experiments are so clear that no justification is needed.

-- David Colquhoun, Lectures on biostatistics, 1971

|cite|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Your post is really a comment, rather than an answer, so I am refraining from upvoting it, but I do wish to thank you for sharing the quotation. There are so many misunderstandings evident in this passage that it would take extensive effort (not to say space) to point out and debunk them all. In one word, though, the counter to these assertions is "efficiency." If everybody had unlimited time and budget we could at least aspire to perform "good experiments." But when resources are limited, it would be foolhardy (as well as costly) to conduct only "final, ... clear" experiments. $\endgroup$ – whuber♦ Apr 8 '15 at 17:11
  • 2
    $\begingroup$ Thanks for your comment, @whuber; I agree with what you are saying. Still, I must add that I do find it appealing to say that ideally experimental data should be so convincing as to render formal hypothesis tests redundant. This is not an unattainable ideal! In my field (where p-values are used a lot), I find that the best papers are convincing without them: e.g. because they present a sequence of several experiments supporting each other, which taken together, obviously cannot be a statistical fluke. Re comment: it was too long for a comment, and I figured it's okay as a CW answer. $\endgroup$ – amoeba says Reinstate Monica Apr 8 '15 at 17:26
  • $\begingroup$ Yes, I understand why it had to be posted as an answer, and therefore did not vote to move it into a comment (which would cut off the last part of the quote). I agree that the ideal is not unattainable in particular cases. I also agree it's a nice ideal to bear in mind. But as a guide to how to design experiments (which is, overall, a discipline of allocating resources), it could be a terrible mistake. (This is certainly debatable.) The suggestion that a "good" experiment would never require statistical methods is, however, one that does not stand up even to cursory examination. $\endgroup$ – whuber♦ Apr 8 '15 at 18:52
  • 1
    $\begingroup$ Perhaps one way of reading that is as saying the initial significance test that suggested a substance stimulates a certain physiological response is no longer relevant by the time you're publishing your investigations into the effects of different kinds of inhibitors on the dose-response curve. $\endgroup$ – Scortchi - Reinstate Monica♦ Apr 27 '16 at 9:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.