Regarding p-values, I am wondering why 1% and 5% seem to be the gold standard for "statistical significance". Why not other values, like 6% or 10%?

Is there a fundamental mathematical reason for this, or is this just a widely held convention?

  • 1
    @Glen_b Your edits always improve things, thanks! – Contango Sep 7 '14 at 10:59
  • What if everyone had 12 fingers? We would be counting base 12, not base 10. And that means that the "1%" would be 1/144 or 0.0069444444. – Contango Dec 24 '15 at 6:41
up vote 75 down vote accepted

If you check the references below you'll find quite a bit of variation in the background, though there are some common elements.

Those numbers are at least partly based on some comments from Fisher, where he said

(while discussing a level of 1/20)

It is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not. Deviations exceeding twice the standard deviation are thus formally regarded as significant

$\quad$ Fisher, R.A. (1925) Statistical Methods for Research Workers, p. 47

On the other hand, he was sometimes more broad:

If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.

$\quad$ Fisher, R.A. (1926) The arrangement of field experiments.
$\quad$ Journal of the Ministry of Agriculture, p. 504

Fisher also used 5% for one of his book's tables - but most of his other tables had a larger variety of significance levels

Some of his comments have suggested more or less strict (i.e. lower or higher alpha levels) approaches in different situations.

That sort of discussion above led to a tendency to produce tables focusing 5% and 1% significance levels (and sometimes with others, like 10%, 2% and 0.5%) for want of any other 'standard' values to use.

However, in this paper, Cowles and Davis suggest that the use of 5% - or something close to it at least - goes back further than Fisher's comment.

In short, our use of 5% (and to a lesser extent 1%) is pretty much arbitrary convention, though clearly a lot of people seem to feel that for many problems they're in the right kind of ballpark.

There's no reason either particular value should be used in general.

Further references:

Dallal, Gerard E. (2012). The Little Handbook of Statistical Practice. - Why 0.05?

Stigler, Stephen (December 2008). "Fisher and the 5% level". Chance 21 (4): 12. available here

(Between them, you get a fair bit of background - it does look like between them there's a good case for thinking significance levels at least in the general ballpark of 5% - say between 2% and 10% - had been more or less in the air for a while.)

I have to give a non-answer (same as here):

"... surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?" (p.1277)

Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276-1284. pdf

The paper contains some more discussion on this issue.

  • 8
    And what about 0.055? :) – nico Apr 10 '13 at 10:50
  • 31
    @nico No one likes 0.055 – Fomite Apr 11 '13 at 4:10

I believe there is some underlying psychology for the 5%. I have to say I don't remember where I picked this up, but here's the exercise I used to do with every undergrad intro stats class.

Imagine a stranger approaches you in a pub and tells you: "I have a biased coin that produces heads more often than tails. Would you like to buy one from me, so that you could bet with your buddies and make money on that?" You hesitantly agree to take a look, and toss the coin say 10 times. Question: how many times does it have to land heads/tails to convince you that it is biased?

Then I take a show of hands: who would be convinced that the coin is biased if the split is 5/5? 4/6? 3/7? 2/8? 1/9? 0/10? Well, the first two or three won't convince anybody, and the last one would convince everybody; 2/8 and 1/9 would convince most people, though. Now, if you look up the binomial table, 2/8 is 5.5%, and 1/9 is 1%. QED.

If anybody is teaching an intro undergrad course right now, I would encourage you to run this exercise, too, and post your results as comments, so that we could accumulate a large body of meta-analysis results and publish them at least in The American Statistician's Teaching Corner. Feel free to vary the $n$ and one-sided vs. two sided conditions!

In another answer, Glen_b quotes Fisher providing the discussion about whether these magic numbers should be modified depending on how serious the problem is, so please don't make it "There's a new treatment for your sister's leukemia, but it would either cure her in 3 months or kill her in 3 days, so let's flip some coins" -- this would look as silly as the infamous xkcd comic that even Andrew Gelman did not like that much.

Speaking of coins and Gelman, TAS had a very curious paper by Gelman and Nolan titled "You can load a die, but you can't bias a coin", putting forth an argument that the coin, flipped in the air or spun on a tabletop, will spend about half of the time heads up, and the other time, tails up, so it is difficult to come up with a physical mechanism to seriously bias a coin. (This clearly was a pub-originated research, as they experimented with beer bottle caps.) On the other hand, loading a die is a relatively easy thing to do, and I gave my students an exercise in that with some 1 cm/half-inch wooden cubes from a local hobby store and sandpaper asking them to load the die, and prove to me it is loaded -- which was an exercise in Pearson $\chi^2$ test for proportions and its power.

  • 3
    Magicians can often control coin flipping. Statistician-mathematician-magician (permute to taste) Persi Diaconis is well known for this (and much, much else). – Nick Cox Aug 21 '13 at 15:12
  • @StasK - A few years ago, I asked a question similar to what's in your second paragraph above. Here's the link: stats.stackexchange.com/questions/7036/… – bill_080 Aug 21 '13 at 15:18
  • bill, you asked about the power, essentially. This question addresses the level of the test. – StasK Aug 21 '13 at 17:25

5% seems to have been rounded from 4.56% by Fisher, corresponding to "the tail areas of the curve beyond the mean plus three or minus three probable errors" (Hurlbert & Lombardi, 2009).

Another element of the story seems to be the reproduction of tables with critical vlaues (Pearson et al., 1990; Lehmann, 1993). Fisher was not given permission by Pearson to use his tables (probably both due to Pearson's marketing of his own publication (Hurlbert & Lombardi, 2009) and the problematic nature of their relationship.

Hurlbert, S. H., & Lombardi, C. M. (2009, October). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. In Annales Zoologici Fennici (Vol. 46, No. 5, pp. 311-349). Finnish Zoological and Botanical Publishing

Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two?. Journal of the American Statistical Association, 88(424), 1242-1249.

Pearson, E. S., Gosset, W. S., Plackett, R. L., & Barnard, G. A. (1990). Student: a statistical biography of William Sealy Gosset. Oxford University Press, USA.

See also: Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.

Hubbard, R., & Lindsay, R. M. (2008). Why P values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18(1), 69-88.

My personal hypothesis is that 0.05 (or 1 in 20) is associated with a t/z value of (very close to) 2. Using 2 is nice, because it's very easy to spot if your result is statistically significant. There aren't other confluences of round numbers.

  • 7
    I doubt this is correct. Of course there are "confluences of round numbers": why not use a critical value of $Z=1$ or $Z=3$, for instance? Moreover, nobody was shying away from making extensive tables of critical values a century ago, so it's difficult to see where the motivation would come from. – whuber Apr 10 '13 at 15:56
  • Because 1 and 3 don't give you something nice, like "1 in 20". It's kind of handy that I can easily spot that my estimate is more than twice my standard error (or not). But I like it as a conspiracy theory. – Jeremy Miles Apr 10 '13 at 17:53
  • 8
    On the contrary, they do give nice numbers! For a Normal distribution the chances are about $1/3$, $1/20$, $1/400$, and $1/16000$ for $z=1,2,3,4$. All these approximations are accurate to better than one significant figure--and the "1 in 20" is the worst of the bunch (1 in 22 would be much closer to the truth). – whuber Apr 10 '13 at 17:58
  • 1
    :) Hmm... good point. But you need to be bounded by what you'd use as a cut-off - 1/3 is a little lax, 1/400 a touch stringent. – Jeremy Miles Apr 10 '13 at 18:07
  • 9
    That's exactly what I'm getting at, Jeremy: the tradition of 5% and 1% is based, at least in part, on a concept of statistical risk ("a little lax" or a "touch stringent") and does not originally derive from any convenient rule of thumb. – whuber Apr 10 '13 at 19:05

Seems to me the answer is more in the game theory of research than in the statistics. Having 1% and 5% burned into the general consciousness means that researchers aren't effectively free to choose significance levels that suit their predispositions. Say we saw a paper with a p-value of .055 and where the significance level had been set at 6% - questions would be asked. 1% and 5% provide a form of credible commitment.

  • 7
    Maybe, but do you think researchers do not manipulate regressions, use repeated testing, etc. to squeeze under the established 5% level for example... – kirk May 20 '13 at 9:12
  • Of course that's possible, and probably happens. But the question was about 1% and 5%. Seems to me like it's an attempt to establish a social convention on when to accept something as significant. These are arbitrary, but they're arbitrary for researchers as a group rather than arbitrary for individual researchers. – conjectures May 21 '13 at 8:44
  • 3
    Agreed, I was just pointing out that having conventional significance levels does not mean questions shouldn't be asked, as you inferred in your post. Just because a paper presents a significant result at a conventional level does not mean it is credible! – kirk May 21 '13 at 13:36
  • Ah, I was using credible in the sense of game theory (or attempting to). As in you make a threat credible if it's not something you can back down from or change your mind about later. In this case individual researchers would have a difficult time alighting on some other arbitrary threshold. – conjectures May 21 '13 at 14:07
  • 1
    What @kirk refers to definitely happens. It's called $p$-hacking. – Nick Stauner Apr 7 '14 at 7:19

The only correct number is .04284731

...which is a flippant response intended to mean that the choice of .05 is essentially arbitrary. I usually just report the p value, rather than what the p value is greater or less than.

"Significance" is a continuous variable, and, in my opinion, discretizing it often does more harm than good. I mean, if p=.13, you've got more confidence than if p=.21 and less than if p=.003

This is an area of hypothesis testing that has always fascinated me. Specifically because one day someone decided on some arbitrary number that dichotomized the testing procedure and since then people rarely question it.

I remember having a lecturer tell us not to put too much faith in the the Staiger and Stock test of instrumental variables (where the F-stat should be above 10 in the first stage regression to avoid weak instrument problems) because the number 10 was a completely arbitrary choice. I remember saying "But is that not what we do with regular hypothesis testing?????"

  • 5
    Is this intended as an answer, @EconStats? It seems more like a comment. Remember that CV is not intended as a discussion forum. Would you mind making the answer w/i this post more salient? – gung Oct 9 '13 at 23:12
  • 1
    Sorry @gung. I guess my point was that, despite some of the evidence provided by the other users, I still think the most likely answer is that we have a decimal based numbering system and it it still being used today to come up with arbitrary numbers for hypothesis tests e.g. the Staiger and Stock F-test that I mentioned. – EconStats Oct 9 '13 at 23:30
  • As the original poster of this question, I believe this definitely qualifies as an answer. Thanks! – Contango Oct 11 '13 at 7:13

Why 1 and 5? Because they feel right.

I'm sure there are studies on the emotional value and cognitive salience of specific numbers, but we can understand the choice of 1 and 5 without having to resort to research.

The people that created today's statistics were born, raised and live in a decimal world. Of course there are non-decimal counting systems, and counting to twelve using the phalanges is possible and has been done, but it is not obvious in the same way as using the fingers is (which are therefore called "digits", like the numbers). And while you (and Fisher) may know about non-decimal counting systems, the decimal system is and has been the predominant counting system your (and Fisher's world) in the past hundred years.

But why are the numbers five and one special? Because both are the most naturally salient divisions of the basic ten: one finger, one hand (or: a half).

You don't even have to go so far as to conceptualize fractions to get from ten to one and five. The one is simply there, just as your finger is simply there. And halving something is an operation much simpler than dividing it into any other proportion. Cutting anything into two parts requires no thinking, while dividing by three or four is already pretty complicated.

Most currenct currency systems have coins and banknotes with values such as 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000. Some currency systems do not have 2, 20 and 200, but almost all have those beginning in 1 and 5. At the same time, most currency systems do not have a coin or banknote that begins in 3, 4, 6, 7, 8 or 9. Interesting, isn't it? But why is that so?

Because you always need either ten of the 1s or two of the 5s (or five of the 2s) to arrive at the next bigger order. Calculating with money is very simple: times ten, or double. Just two kinds of operations. Every coin that you have is either half or a tenth of the next order coin. Those numbers multiply and add up easily and well.

So the 1 and 5 have been deeply ingrained, from their earliest childhood on, into Fisher and whoever else chose the significance levels as the most straightforward, most simple, most basic divisions of 10. Any other number needs an argument for it, while these numbers are simply there.

In the absence of an objective way to calculate the appropriate significance level for every individual data set, the one and five just feel right.

  • "without having to resort to research." While I think the answer is nice, this puts it firmly into opinion territory. It would lend much credibility and would make the answer more authoritative if there were sources to back this up. – Momo Jun 5 '16 at 12:59

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.