# Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding s, I am wondering why $$1$$% and $$5$$% seem to be the gold standard for "statistical significance". Why not other values, like $$6$$% or $$10$$%?

Is there a fundamental mathematical reason for this, or is this just a widely held convention?

• What if everyone had 12 fingers? We would be counting base 12, not base 10. And that means that the "1%" would be 1/144 or 0.0069444444. Dec 24, 2015 at 6:41

## 10 Answers

If you check the references below you'll find quite a bit of variation in the background, though there are some common elements.

Those numbers are at least partly based on some comments from Fisher, where he said

(while discussing a level of 1/20)

It is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not. Deviations exceeding twice the standard deviation are thus formally regarded as significant

$\quad$ Fisher, R.A. (1925) Statistical Methods for Research Workers, p. 47

On the other hand, he was sometimes more broad:

If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.

$\quad$ Fisher, R.A. (1926) The arrangement of field experiments.
$\quad$ Journal of the Ministry of Agriculture, p. 504

Fisher also used 5% for one of his book's tables - but most of his other tables had a larger variety of significance levels

Some of his comments have suggested more or less strict (i.e. lower or higher alpha levels) approaches in different situations.

That sort of discussion above led to a tendency to produce tables focusing 5% and 1% significance levels (and sometimes with others, like 10%, 2% and 0.5%) for want of any other 'standard' values to use.

However, in this paper, Cowles and Davis suggest that the use of 5% - or something close to it at least - goes back further than Fisher's comment.

In short, our use of 5% (and to a lesser extent 1%) is pretty much arbitrary convention, though clearly a lot of people seem to feel that for many problems they're in the right kind of ballpark.

There's no reason either particular value should be used in general.

Further references:

Dallal, Gerard E. (2012). The Little Handbook of Statistical Practice. - Why 0.05?

Stigler, Stephen (December 2008). "Fisher and the 5% level". Chance 21 (4): 12. available here

(Between them, you get a fair bit of background - it does look like between them there's a good case for thinking significance levels at least in the general ballpark of 5% - say between 2% and 10% - had been more or less in the air for a while.)

I have to give a non-answer (same as here):

"... surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?" (p.1277)

Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276-1284. pdf

The paper contains some more discussion on this issue.

• And what about 0.055? :)
– nico
Apr 10, 2013 at 10:50
• @nico No one likes 0.055 Apr 11, 2013 at 4:10

I believe there is some underlying psychology for the 5%. I have to say I don't remember where I picked this up, but here's the exercise I used to do with every undergrad intro stats class.

Imagine a stranger approaches you in a pub and tells you: "I have a biased coin that produces heads more often than tails. Would you like to buy one from me, so that you could bet with your buddies and make money on that?" You hesitantly agree to take a look, and toss the coin say 10 times. Question: how many times does it have to land heads/tails to convince you that it is biased?

Then I take a show of hands: who would be convinced that the coin is biased if the split is 5/5? 4/6? 3/7? 2/8? 1/9? 0/10? Well, the first two or three won't convince anybody, and the last one would convince everybody; 2/8 and 1/9 would convince most people, though. Now, if you look up the binomial table, 2/8 is 5.5%, and 1/9 is 1%. QED.

If anybody is teaching an intro undergrad course right now, I would encourage you to run this exercise, too, and post your results as comments, so that we could accumulate a large body of meta-analysis results and publish them at least in The American Statistician's Teaching Corner. Feel free to vary the $n$ and one-sided vs. two sided conditions!

In another answer, Glen_b quotes Fisher providing the discussion about whether these magic numbers should be modified depending on how serious the problem is, so please don't make it "There's a new treatment for your sister's leukemia, but it would either cure her in 3 months or kill her in 3 days, so let's flip some coins" -- this would look as silly as the infamous xkcd comic that even Andrew Gelman did not like that much.

Speaking of coins and Gelman, TAS had a very curious paper by Gelman and Nolan titled "You can load a die, but you can't bias a coin", putting forth an argument that the coin, flipped in the air or spun on a tabletop, will spend about half of the time heads up, and the other time, tails up, so it is difficult to come up with a physical mechanism to seriously bias a coin. (This clearly was a pub-originated research, as they experimented with beer bottle caps.) On the other hand, loading a die is a relatively easy thing to do, and I gave my students an exercise in that with some 1 cm/half-inch wooden cubes from a local hobby store and sandpaper asking them to load the die, and prove to me it is loaded -- which was an exercise in Pearson $\chi^2$ test for proportions and its power.

• Magicians can often control coin flipping. Statistician-mathematician-magician (permute to taste) Persi Diaconis is well known for this (and much, much else). Aug 21, 2013 at 15:12
• @StasK - A few years ago, I asked a question similar to what's in your second paragraph above. Here's the link: stats.stackexchange.com/questions/7036/… Aug 21, 2013 at 15:18
• bill, you asked about the power, essentially. This question addresses the level of the test. Aug 21, 2013 at 17:25
• Nice answer!!! I was just (unintentionally) annoying Andrew Gelman on his blog last night! Here's another scholarly journal article about flipping coins, The Probability of Heads circa 1986, same source publication. I wish I could read more than the first page but alas, paywall. Nov 16, 2022 at 13:12

5% seems to have been rounded from 4.56% by Fisher, corresponding to "the tail areas of the curve beyond the mean plus three or minus three probable errors" (Hurlbert & Lombardi, 2009).

Another element of the story seems to be the reproduction of tables with critical vlaues (Pearson et al., 1990; Lehmann, 1993). Fisher was not given permission by Pearson to use his tables (probably both due to Pearson's marketing of his own publication (Hurlbert & Lombardi, 2009) and the problematic nature of their relationship.

Hurlbert, S. H., & Lombardi, C. M. (2009, October). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. In Annales Zoologici Fennici (Vol. 46, No. 5, pp. 311-349). Finnish Zoological and Botanical Publishing

Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two?. Journal of the American Statistical Association, 88(424), 1242-1249.

Pearson, E. S., Plackett, R. L., & Barnard, G. A. (1990). Student: a statistical biography of William Sealy Gosset. Oxford University Press, USA.

See also: Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.

Hubbard, R., & Lindsay, R. M. (2008). Why P values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18(1), 69-88.

Seems to me the answer is more in the game theory of research than in the statistics. Having 1% and 5% burned into the general consciousness means that researchers aren't effectively free to choose significance levels that suit their predispositions. Say we saw a paper with a p-value of .055 and where the significance level had been set at 6% - questions would be asked. 1% and 5% provide a form of credible commitment.

• Maybe, but do you think researchers do not manipulate regressions, use repeated testing, etc. to squeeze under the established 5% level for example...
– kirk
May 20, 2013 at 9:12
• Of course that's possible, and probably happens. But the question was about 1% and 5%. Seems to me like it's an attempt to establish a social convention on when to accept something as significant. These are arbitrary, but they're arbitrary for researchers as a group rather than arbitrary for individual researchers. May 21, 2013 at 8:44
• Agreed, I was just pointing out that having conventional significance levels does not mean questions shouldn't be asked, as you inferred in your post. Just because a paper presents a significant result at a conventional level does not mean it is credible!
– kirk
May 21, 2013 at 13:36
• Ah, I was using credible in the sense of game theory (or attempting to). As in you make a threat credible if it's not something you can back down from or change your mind about later. In this case individual researchers would have a difficult time alighting on some other arbitrary threshold. May 21, 2013 at 14:07
• What @kirk refers to definitely happens. It's called $p$-hacking. Apr 7, 2014 at 7:19

The only correct number is .04284731

...which is a flippant response intended to mean that the choice of .05 is essentially arbitrary. I usually just report the p value, rather than what the p value is greater or less than.

"Significance" is a continuous variable, and, in my opinion, discretizing it often does more harm than good. I mean, if p=.13, you've got more confidence than if p=.21 and less than if p=.003

• Well, in the times of tables one were more or less forced to discretize ... since tables are used in teaching, this continues ... Dec 18, 2018 at 11:46
• @kjetilbhalvorsen well the table makers clearly erred in not choosing .04284731 for their critical values. Dec 18, 2018 at 15:58

My personal hypothesis is that 0.05 (or 1 in 20) is associated with a t/z value of (very close to) 2. Using 2 is nice, because it's very easy to spot if your result is statistically significant. There aren't other confluences of round numbers.

• I doubt this is correct. Of course there are "confluences of round numbers": why not use a critical value of $Z=1$ or $Z=3$, for instance? Moreover, nobody was shying away from making extensive tables of critical values a century ago, so it's difficult to see where the motivation would come from.
– whuber
Apr 10, 2013 at 15:56
• On the contrary, they do give nice numbers! For a Normal distribution the chances are about $1/3$, $1/20$, $1/400$, and $1/16000$ for $z=1,2,3,4$. All these approximations are accurate to better than one significant figure--and the "1 in 20" is the worst of the bunch (1 in 22 would be much closer to the truth).
– whuber
Apr 10, 2013 at 17:58
• :) Hmm... good point. But you need to be bounded by what you'd use as a cut-off - 1/3 is a little lax, 1/400 a touch stringent. Apr 10, 2013 at 18:07
• That's exactly what I'm getting at, Jeremy: the tradition of 5% and 1% is based, at least in part, on a concept of statistical risk ("a little lax" or a "touch stringent") and does not originally derive from any convenient rule of thumb.
– whuber
Apr 10, 2013 at 19:05
• @whuber Using $Z=1$ gives approximately $1/{\pi}$, and you can't get much more round than that! Oct 16, 2019 at 15:42

A recent article sheds some light on the arbitrariness of $$p$$-values; the selection of two thresholds was motivated, at least in part, as a work-around to a dispute over publishing rights. Briefly, Fisher sought to use continuous-valued $$p$$-values as a characterization of strength of evidence. But he would not be able to publish accompanying tables to aid in their computation because of a copyright claim. To avoid copyright infringement, Fisher dichotomized $$p$$-values into "significant" and "non-significant." This meant he could publish critical values alone, without reproducing the entire tables.

An awareness of the history of $$p$$-values might help deflate their swollen stature and encourage more judicious use. We were surprised to learn, in the course of writing this article, that the $$p < 0.05$$ cutoff was established as a competitive response to a disagreement over book royalties between two foundational statisticians. In the early 1920s, Kendall Pearson [sic -- I believe this is a typo for Karl Pearson, a prominent statistician who published mathematical and statistical tables in the 1920s], whose income depended on the sale of extensive statistical tables, was unwilling to allow Ronald A. Fisher to use them in his new book. To work around this barrier, Fisher created a method of inference based on only two values: $$p$$-values of 0.05 and 0.01 (Hurlbert and Lombardi, 2009). Fisher himself later admitted that Perason's more continuous method of inference was better than his binary approach: "no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects [null] hypotheses; he rather gives his mind to each particular case in the light of his evidence and ideas" (Hurlbert and Lombardi, 2009; 316). A fair interpretation of this history is that we use $$p$$-values at least in part because a statistician in the 1920s was afraid that sharing his work would undermine his income (Hurlbert and Lombardi, 2009). Following Fisher, we recommend that authors report $$p$$-values and refrain from emphasizing thresholds.

from Brent Goldfarb and Andrew W. King. "Scientific Apophenia in Strategic Management Research: Significance & Mistaken Inference." Strategic Management Journal, vol. 37, no. 1, Wiley, 2016, pp. 167–76.

Now, this passage does not answer the titular question "Why did Fisher choose 0.05 and 0.01 instead of 0.06 or 0.1?" After all, Fisher could have chosen to publish his book using 0.06 and 0.1 in place of 0.05 and 0.01 (or indeed he could have chosen any other probabilities). However, this passage does show that Fisher understood that the choice was arbitrary in its very nature, and that a single threshold for adjudicating all statistical inference is unsuitable.

We might imagine a dramatically different statistical practice around hypothesis testing and inference if Fisher were instead able to publish Pearson's statistical tables! And while we're imagining some alternative worlds, we might also explore whether "significance & non-significance" are essential concepts for inference.

Null hypothesis testing has no inherent requirement that $$\alpha$$ be specified, or that the "significant/non-significant" terminology be adopted. Fisher may have been impelled to those conventions, however, not only by historical antecedents but also by a very practical and personal obstacle. Kendall (1963) relates that "He [Fisher] himself told me that when he was writing Statistical Methods for Research Workers he applied to Pearson for permission to reproduce Elderton's tables of chi-squared and that it was refused. This was perhaps not simply a personal matter because the hard struggle which Pearson had for long experienced in obtaining funds for printing and publishing statistical tables had made him most unwilling to grant anyone permission to reproduce. He was afraid of the effect on sales of his Tables for Statisticians and Biometricians [K. Pearson 1914] on which he relied to secure money for further table publication. It seems, however, to have been this refusal which first directed Fisher's thoughts towards the alternative form of tabulation with quantiles as argument, a form which he subsequently adopted for all his tables and which has become common practice."

This is what Fisher referred to when eh explained the absence from his book of more extended tables "owing to copyright restrictions" (Fisher 1925: 78, 1958: 79). Fisher did not invent the "significant/non-significant" dichotomy, but his books and novel tabulations of critical values of test statistics played a large role in its rapid and wide dissemination.

from Hurlbert, Stuart H., and Celia M. Lombardi. 2009. “Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian.” Annales Zoologici Fennici 46 (5): 311–49.

• Karl Pearson edited a book of tables (Tables for Statisticians and Biometricians) published in 1914 by CUP, based on various tables that had been published in Biometrika from 1901 onward. There are multiple versions of it on the Internet archive. But the normal tables there would presumably not be an issue, since much older normal tables had been published. Maybe the chi-squared tables could have been an issue. Mar 27, 2022 at 7:53

This is an area of hypothesis testing that has always fascinated me. Specifically because one day someone decided on some arbitrary number that dichotomized the testing procedure and since then people rarely question it.

I remember having a lecturer tell us not to put too much faith in the the Staiger and Stock test of instrumental variables (where the F-stat should be above 10 in the first stage regression to avoid weak instrument problems) because the number 10 was a completely arbitrary choice. I remember saying "But is that not what we do with regular hypothesis testing?????"

• Is this intended as an answer, @EconStats? It seems more like a comment. Remember that CV is not intended as a discussion forum. Would you mind making the answer w/i this post more salient? Oct 9, 2013 at 23:12
• Sorry @gung. I guess my point was that, despite some of the evidence provided by the other users, I still think the most likely answer is that we have a decimal based numbering system and it it still being used today to come up with arbitrary numbers for hypothesis tests e.g. the Staiger and Stock F-test that I mentioned. Oct 9, 2013 at 23:30
• As the original poster of this question, I believe this definitely qualifies as an answer. Thanks! Oct 11, 2013 at 7:13

Why 1 and 5? Because they feel right.

I'm sure there are studies on the emotional value and cognitive salience of specific numbers, but we can understand the choice of 1 and 5 without having to resort to research.

The people that created today's statistics were born, raised and live in a decimal world. Of course there are non-decimal counting systems, and counting to twelve using the phalanges is possible and has been done, but it is not obvious in the same way as using the fingers is (which are therefore called "digits", like the numbers). And while you (and Fisher) may know about non-decimal counting systems, the decimal system is and has been the predominant counting system your (and Fisher's world) in the past hundred years.

But why are the numbers five and one special? Because both are the most naturally salient divisions of the basic ten: one finger, one hand (or: a half).

You don't even have to go so far as to conceptualize fractions to get from ten to one and five. The one is simply there, just as your finger is simply there. And halving something is an operation much simpler than dividing it into any other proportion. Cutting anything into two parts requires no thinking, while dividing by three or four is already pretty complicated.

Most currenct currency systems have coins and banknotes with values such as 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000. Some currency systems do not have 2, 20 and 200, but almost all have those beginning in 1 and 5. At the same time, most currency systems do not have a coin or banknote that begins in 3, 4, 6, 7, 8 or 9. Interesting, isn't it? But why is that so?

Because you always need either ten of the 1s or two of the 5s (or five of the 2s) to arrive at the next bigger order. Calculating with money is very simple: times ten, or double. Just two kinds of operations. Every coin that you have is either half or a tenth of the next order coin. Those numbers multiply and add up easily and well.

So the 1 and 5 have been deeply ingrained, from their earliest childhood on, into Fisher and whoever else chose the significance levels as the most straightforward, most simple, most basic divisions of 10. Any other number needs an argument for it, while these numbers are simply there.

In the absence of an objective way to calculate the appropriate significance level for every individual data set, the one and five just feel right.

• "without having to resort to research." While I think the answer is nice, this puts it firmly into opinion territory. It would lend much credibility and would make the answer more authoritative if there were sources to back this up.
– Momo
Jun 5, 2016 at 12:59