Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%?
Is there a fundamental mathematical reason for this, or is this just a widely held convention?
If you check the references below you'll find quite a bit of variation in the background, though there are some common elements.
Those numbers are at least partly based on some comments from Fisher, where he said
(while discussing a level of 1/20)
It is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not. Deviations exceeding twice the standard deviation are thus formally regarded as significant
$\quad$ Fisher, R.A. (1925) Statistical Methods for Research Workers, p. 47
On the other hand, he was sometimes more broad:
If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.
$\quad$ Fisher, R.A. (1926)
The arrangement of field experiments.
$\quad$ Journal
of the Ministry of Agriculture, p. 504
Fisher also used 5% for one of his book's tables - but most of his other tables had a larger variety of significance levels
Some of his comments have suggested more or less strict (i.e. lower or higher alpha levels) approaches in different situations.
That sort of discussion above led to a tendency to produce tables focusing 5% and 1% significance levels (and sometimes with others, like 10%, 2% and 0.5%) for want of any other 'standard' values to use.
However, in this paper, Cowles and Davis suggest that the use of 5% - or something close to it at least - goes back further than Fisher's comment.
In short, our use of 5% (and to a lesser extent 1%) is pretty much arbitrary convention, though clearly a lot of people seem to feel that for many problems they're in the right kind of ballpark.
There's no reason either particular value should be used in general.
Further references:
Dallal, Gerard E. (2012). The Little Handbook of Statistical Practice. - Why 0.05?
Stigler, Stephen (December 2008). "Fisher and the 5% level". Chance 21 (4): 12. available here
(Between them, you get a fair bit of background - it does look like between them there's a good case for thinking significance levels at least in the general ballpark of 5% - say between 2% and 10% - had been more or less in the air for a while.)
I have to give a non-answer (same as here):
"... surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?" (p.1277)
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), 1276-1284. pdf
The paper contains some more discussion on this issue.
I believe there is some underlying psychology for the 5%. I have to say I don't remember where I picked this up, but here's the exercise I used to do with every undergrad intro stats class.
Imagine a stranger approaches you in a pub and tells you: "I have a biased coin that produces heads more often than tails. Would you like to buy one from me, so that you could bet with your buddies and make money on that?" You hesitantly agree to take a look, and toss the coin say 10 times. Question: how many times does it have to land heads/tails to convince you that it is biased?
Then I take a show of hands: who would be convinced that the coin is biased if the split is 5/5? 4/6? 3/7? 2/8? 1/9? 0/10? Well, the first two or three won't convince anybody, and the last one would convince everybody; 2/8 and 1/9 would convince most people, though. Now, if you look up the binomial table, 2/8 is 5.5%, and 1/9 is 1%. QED.
If anybody is teaching an intro undergrad course right now, I would encourage you to run this exercise, too, and post your results as comments, so that we could accumulate a large body of meta-analysis results and publish them at least in The American Statistician's Teaching Corner. Feel free to vary the $n$ and one-sided vs. two sided conditions!
In another answer, Glen_b quotes Fisher providing the discussion about whether these magic numbers should be modified depending on how serious the problem is, so please don't make it "There's a new treatment for your sister's leukemia, but it would either cure her in 3 months or kill her in 3 days, so let's flip some coins" -- this would look as silly as the infamous xkcd comic that even Andrew Gelman did not like that much.
Speaking of coins and Gelman, TAS had a very curious paper by Gelman and Nolan titled "You can load a die, but you can't bias a coin", putting forth an argument that the coin, flipped in the air or spun on a tabletop, will spend about half of the time heads up, and the other time, tails up, so it is difficult to come up with a physical mechanism to seriously bias a coin. (This clearly was a pub-originated research, as they experimented with beer bottle caps.) On the other hand, loading a die is a relatively easy thing to do, and I gave my students an exercise in that with some 1 cm/half-inch wooden cubes from a local hobby store and sandpaper asking them to load the die, and prove to me it is loaded -- which was an exercise in Pearson $\chi^2$ test for proportions and its power.
5% seems to have been rounded from 4.56% by Fisher, corresponding to "the tail areas of the curve beyond the mean plus three or minus three probable errors" (Hurlbert & Lombardi, 2009).
Another element of the story seems to be the reproduction of tables with critical vlaues (Pearson et al., 1990; Lehmann, 1993). Fisher was not given permission by Pearson to use his tables (probably both due to Pearson's marketing of his own publication (Hurlbert & Lombardi, 2009) and the problematic nature of their relationship.
Hurlbert, S. H., & Lombardi, C. M. (2009, October). Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian. In Annales Zoologici Fennici (Vol. 46, No. 5, pp. 311-349). Finnish Zoological and Botanical Publishing
Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two?. Journal of the American Statistical Association, 88(424), 1242-1249.
Pearson, E. S., Plackett, R. L., & Barnard, G. A. (1990). Student: a statistical biography of William Sealy Gosset. Oxford University Press, USA.
See also: Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.
Hubbard, R., & Lindsay, R. M. (2008). Why P values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18(1), 69-88.
Seems to me the answer is more in the game theory of research than in the statistics. Having 1% and 5% burned into the general consciousness means that researchers aren't effectively free to choose significance levels that suit their predispositions. Say we saw a paper with a p-value of .055 and where the significance level had been set at 6% - questions would be asked. 1% and 5% provide a form of credible commitment.
The only correct number is .04284731
...which is a flippant response intended to mean that the choice of .05 is essentially arbitrary. I usually just report the p value, rather than what the p value is greater or less than.
"Significance" is a continuous variable, and, in my opinion, discretizing it often does more harm than good. I mean, if p=.13, you've got more confidence than if p=.21 and less than if p=.003
My personal hypothesis is that 0.05 (or 1 in 20) is associated with a t/z value of (very close to) 2. Using 2 is nice, because it's very easy to spot if your result is statistically significant. There aren't other confluences of round numbers.
A recent article sheds some light on the arbitrariness of $p$-values; the selection of two thresholds was motivated, at least in part, as a work-around to a dispute over publishing rights. Briefly, Fisher sought to use continuous-valued $p$-values as a characterization of strength of evidence. But he would not be able to publish accompanying tables to aid in their computation because of a copyright claim. To avoid copyright infringement, Fisher dichotomized $p$-values into "significant" and "non-significant." This meant he could publish critical values alone, without reproducing the entire tables.
An awareness of the history of $p$-values might help deflate their swollen stature and encourage more judicious use. We were surprised to learn, in the course of writing this article, that the $p < 0.05$ cutoff was established as a competitive response to a disagreement over book royalties between two foundational statisticians. In the early 1920s, Kendall Pearson [sic -- I believe this is a typo for Karl Pearson, a prominent statistician who published mathematical and statistical tables in the 1920s], whose income depended on the sale of extensive statistical tables, was unwilling to allow Ronald A. Fisher to use them in his new book. To work around this barrier, Fisher created a method of inference based on only two values: $p$-values of 0.05 and 0.01 (Hurlbert and Lombardi, 2009). Fisher himself later admitted that Perason's more continuous method of inference was better than his binary approach: "no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects [null] hypotheses; he rather gives his mind to each particular case in the light of his evidence and ideas" (Hurlbert and Lombardi, 2009; 316). A fair interpretation of this history is that we use $p$-values at least in part because a statistician in the 1920s was afraid that sharing his work would undermine his income (Hurlbert and Lombardi, 2009). Following Fisher, we recommend that authors report $p$-values and refrain from emphasizing thresholds.
from Brent Goldfarb and Andrew W. King. "Scientific Apophenia in Strategic Management Research: Significance & Mistaken Inference." Strategic Management Journal, vol. 37, no. 1, Wiley, 2016, pp. 167–76.
Now, this passage does not answer the titular question "Why did Fisher choose 0.05 and 0.01 instead of 0.06 or 0.1?" After all, Fisher could have chosen to publish his book using 0.06 and 0.1 in place of 0.05 and 0.01 (or indeed he could have chosen any other probabilities). However, this passage does show that Fisher understood that the choice was arbitrary in its very nature, and that a single threshold for adjudicating all statistical inference is unsuitable.
We might imagine a dramatically different statistical practice around hypothesis testing and inference if Fisher were instead able to publish Pearson's statistical tables! And while we're imagining some alternative worlds, we might also explore whether "significance & non-significance" are essential concepts for inference.
Null hypothesis testing has no inherent requirement that $\alpha$ be specified, or that the "significant/non-significant" terminology be adopted. Fisher may have been impelled to those conventions, however, not only by historical antecedents but also by a very practical and personal obstacle. Kendall (1963) relates that "He [Fisher] himself told me that when he was writing Statistical Methods for Research Workers he applied to Pearson for permission to reproduce Elderton's tables of chi-squared and that it was refused. This was perhaps not simply a personal matter because the hard struggle which Pearson had for long experienced in obtaining funds for printing and publishing statistical tables had made him most unwilling to grant anyone permission to reproduce. He was afraid of the effect on sales of his Tables for Statisticians and Biometricians [K. Pearson 1914] on which he relied to secure money for further table publication. It seems, however, to have been this refusal which first directed Fisher's thoughts towards the alternative form of tabulation with quantiles as argument, a form which he subsequently adopted for all his tables and which has become common practice."
This is what Fisher referred to when eh explained the absence from his book of more extended tables "owing to copyright restrictions" (Fisher 1925: 78, 1958: 79). Fisher did not invent the "significant/non-significant" dichotomy, but his books and novel tabulations of critical values of test statistics played a large role in its rapid and wide dissemination.
from Hurlbert, Stuart H., and Celia M. Lombardi. 2009. “Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian.” Annales Zoologici Fennici 46 (5): 311–49.
This is an area of hypothesis testing that has always fascinated me. Specifically because one day someone decided on some arbitrary number that dichotomized the testing procedure and since then people rarely question it.
I remember having a lecturer tell us not to put too much faith in the the Staiger and Stock test of instrumental variables (where the F-stat should be above 10 in the first stage regression to avoid weak instrument problems) because the number 10 was a completely arbitrary choice. I remember saying "But is that not what we do with regular hypothesis testing?????"
Why 1 and 5? Because they feel right.
I'm sure there are studies on the emotional value and cognitive salience of specific numbers, but we can understand the choice of 1 and 5 without having to resort to research.
The people that created today's statistics were born, raised and live in a decimal world. Of course there are non-decimal counting systems, and counting to twelve using the phalanges is possible and has been done, but it is not obvious in the same way as using the fingers is (which are therefore called "digits", like the numbers). And while you (and Fisher) may know about non-decimal counting systems, the decimal system is and has been the predominant counting system your (and Fisher's world) in the past hundred years.
But why are the numbers five and one special? Because both are the most naturally salient divisions of the basic ten: one finger, one hand (or: a half).
You don't even have to go so far as to conceptualize fractions to get from ten to one and five. The one is simply there, just as your finger is simply there. And halving something is an operation much simpler than dividing it into any other proportion. Cutting anything into two parts requires no thinking, while dividing by three or four is already pretty complicated.
Most currenct currency systems have coins and banknotes with values such as 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000. Some currency systems do not have 2, 20 and 200, but almost all have those beginning in 1 and 5. At the same time, most currency systems do not have a coin or banknote that begins in 3, 4, 6, 7, 8 or 9. Interesting, isn't it? But why is that so?
Because you always need either ten of the 1s or two of the 5s (or five of the 2s) to arrive at the next bigger order. Calculating with money is very simple: times ten, or double. Just two kinds of operations. Every coin that you have is either half or a tenth of the next order coin. Those numbers multiply and add up easily and well.
So the 1 and 5 have been deeply ingrained, from their earliest childhood on, into Fisher and whoever else chose the significance levels as the most straightforward, most simple, most basic divisions of 10. Any other number needs an argument for it, while these numbers are simply there.
In the absence of an objective way to calculate the appropriate significance level for every individual data set, the one and five just feel right.
