What is significant?

Question

There is a widespread notion that a p-value less than 0.05 is to be considered statistically significant, and above, not so.

I understand that the "0.05" was an arbitrary number, albeit arbitrated by a very smart man. In principle, it could've easily been 0.1, or 0.01. Most of the widespread support is just people using whatever everyone else is using.

This question asks, in effect IMO, whether it's okay to pick something like 0.07 as your threshold instead. Of course, probably it is preferable to select the threshold before you do your experiment, so that you don't end up in the silly situation where you results are significant because you happen to have picked your significance threshold to be whatever makes them significant.

But, once again, the 0.05 was arbitrary. It wasn't even the good kind of arbitrary, where excellent reasons exist for choosing it over all other alternatives, it is just technically not unique. AFAIK, it wasn't even a particularly suitable threshold, and it probably isn't optimal in many cases (for instance, I think it's fair to say that for multiple hypothesis tests, 0.05 is not appropriate). If Fisher ever made some argument why it should be 0.05 and not something else, I'm not aware of it.

But today, we have the benefit of an ocean of knowledge and experience in statistics, as well as having learned a great deal of thinks about how large research communities behave in the long term. For instance, the famous Ioannidis paper, while not concerned with the significance threshold per se, seems like it would have been not wholly irrelevant to Fisher's thinking had he known about it.

Given this, is it possible for us to do better than Fisher? Is there a persuasive argument to be made in favor for adoption of a different threshold? Is there a technique that can be applied to select an appropriate threshold for an experiment that one is planning?

Answer 1

As you noted it yourself, the choice of 0.05 as the threshold was purely arbitrary. The reason why it became the default choice in many sciences is because it is convenient. Towards the end of his career, Fisher (1971) also argued that

"It is usual and convenient for experimenters to take 5% as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means, to eliminate from further discussion the greater part of fluctuations which chance causes have introduced into their experimental results (p.13)"

However, technically speaking, every researcher is free to choose the threshold themselves. There is no reason why you cannot choose 0.06 or 0.10 as the threshold if that makes sense for your research question. The only problem comes when you try to publish the results- most editors and reviewers will not be happy about it and your paper will almost certainly not get published. In this sense, in the real world you can get away with not using 0.05 only if you choose a smaller level (e.g. 0.01).

Recently, there have been such calls for setting a smaller threshold (0.005 or 0.001; Johnson, 2013), supposedly to improve the replicability of research findings. However, this also comes at a price because of the loss in statistical power and the (much) larger sample sizes that would be needed.

So, to answer your question, I do not think that there is a good way of finding the optimal threshold (probably because there isn't one). In an ideal world, different thresholds would be used for different types of research problems, but as mentioned previously, this wouldn't work in reality. I agree with HEITZ that researchers would gain a lot if they move to other ways of making statistical inference, such as effect size estimation or Bayesian inference.

References:

Fisher, R. A. (1971). The design of experiments(9th ed.). New York: Hafner publishing company.

Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313-19317.

Answer 2

From my perspective, there would be much more to gain from moving the field(s) away from NHST entirely in favor of effect sizes. Think of it this way - suppose 100 studies are conducted on the same problem by 100 research groups, using different methodologies. All groups use NHST with alpha = 0.05. 5 of those studies would be expected to be Type I errors. Ok, fine, we knew that, but using only p values, we have no idea which studies had the stronger manipulations. What am I, a consumer of research, then to do with this information? Should I weight all the 'significant' studies equally? Probably not. Had all the groups instead relied on effect sizes, we would have a quantitative ranking (more or less, discounting vast differences in experimental design).