3
$\begingroup$

There is a widespread notion that a p-value less than 0.05 is to be considered statistically significant, and above, not so.

I understand that the "0.05" was an arbitrary number, albeit arbitrated by a very smart man. In principle, it could've easily been 0.1, or 0.01. Most of the widespread support is just people using whatever everyone else is using.

This question asks, in effect IMO, whether it's okay to pick something like 0.07 as your threshold instead. Of course, probably it is preferable to select the threshold before you do your experiment, so that you don't end up in the silly situation where you results are significant because you happen to have picked your significance threshold to be whatever makes them significant.

But, once again, the 0.05 was arbitrary. It wasn't even the good kind of arbitrary, where excellent reasons exist for choosing it over all other alternatives, it is just technically not unique. AFAIK, it wasn't even a particularly suitable threshold, and it probably isn't optimal in many cases (for instance, I think it's fair to say that for multiple hypothesis tests, 0.05 is not appropriate). If Fisher ever made some argument why it should be 0.05 and not something else, I'm not aware of it.

But today, we have the benefit of an ocean of knowledge and experience in statistics, as well as having learned a great deal of thinks about how large research communities behave in the long term. For instance, the famous Ioannidis paper, while not concerned with the significance threshold per se, seems like it would have been not wholly irrelevant to Fisher's thinking had he known about it.

Given this, is it possible for us to do better than Fisher? Is there a persuasive argument to be made in favor for adoption of a different threshold? Is there a technique that can be applied to select an appropriate threshold for an experiment that one is planning?

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ To quote the man I presume you mean (Fisher): "If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance" ... ctd $\endgroup$ – Glen_b -Reinstate Monica Sep 18 '15 at 0:47
  • 2
    $\begingroup$ ctd... Your characterization of 0.05 as "arbitrated" is something of an oversimplicification since his view is (at least sometimes) rather more nuanced, and he often seems to regard 5% as more of an "upper limit" of the kind of significance level he'd consider, and in a scientific context he'd look to replicate and achieve it repeatedly. (i.e. corresponding to an overall level that would be considerably lower). At other times he does specifically give 5%, but it's often clearly personal choice. See stats.stackexchange.com/questions/55691/… $\endgroup$ – Glen_b -Reinstate Monica Sep 18 '15 at 0:48
2
$\begingroup$

As you noted it yourself, the choice of 0.05 as the threshold was purely arbitrary. The reason why it became the default choice in many sciences is because it is convenient. Towards the end of his career, Fisher (1971) also argued that

"It is usual and convenient for experimenters to take 5% as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means, to eliminate from further discussion the greater part of fluctuations which chance causes have introduced into their experimental results (p.13)"

However, technically speaking, every researcher is free to choose the threshold themselves. There is no reason why you cannot choose 0.06 or 0.10 as the threshold if that makes sense for your research question. The only problem comes when you try to publish the results- most editors and reviewers will not be happy about it and your paper will almost certainly not get published. In this sense, in the real world you can get away with not using 0.05 only if you choose a smaller level (e.g. 0.01).

Recently, there have been such calls for setting a smaller threshold (0.005 or 0.001; Johnson, 2013), supposedly to improve the replicability of research findings. However, this also comes at a price because of the loss in statistical power and the (much) larger sample sizes that would be needed.

So, to answer your question, I do not think that there is a good way of finding the optimal threshold (probably because there isn't one). In an ideal world, different thresholds would be used for different types of research problems, but as mentioned previously, this wouldn't work in reality. I agree with HEITZ that researchers would gain a lot if they move to other ways of making statistical inference, such as effect size estimation or Bayesian inference.

References:

Fisher, R. A. (1971). The design of experiments(9th ed.). New York: Hafner publishing company.

Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313-19317.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Agreeing with this response, the original reseacher having taken sufficient time and effort to obtain a level of the arbitrary 0.05 or less makes it worthwhile for other people to try to replicate it. $\endgroup$ – Robert Jones Dec 20 '15 at 20:46
  • $\begingroup$ on reflection, it is arguable that negative results should also be published or at least stored in an openly searchable repository as per the discussions arising from the Ioannides paper $\endgroup$ – Robert Jones Dec 20 '15 at 21:19
0
$\begingroup$

From my perspective, there would be much more to gain from moving the field(s) away from NHST entirely in favor of effect sizes. Think of it this way - suppose 100 studies are conducted on the same problem by 100 research groups, using different methodologies. All groups use NHST with alpha = 0.05. 5 of those studies would be expected to be Type I errors. Ok, fine, we knew that, but using only p values, we have no idea which studies had the stronger manipulations. What am I, a consumer of research, then to do with this information? Should I weight all the 'significant' studies equally? Probably not. Had all the groups instead relied on effect sizes, we would have a quantitative ranking (more or less, discounting vast differences in experimental design).

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.