I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a common standard?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
3
My opinion is that it does (and should) not depend on the field of study. For example, you may well work at a lower significance level than $p<0.001$ if, for example, you are trying to replicate a study with historical or well-established results (I can think of several studies on the Stroop effect, which had led to some controversies in the past few years). That amounts to consider a lower "threshold" within the classical Neyman-Pearson framework for testing hypothesis. However, statistical and practical (or substantive) significance is another matter.
Sidenote. The "star system" seems to have dominated scientific inquiries as early as the 70's, but see The Earth Is Round (p < .05), by J. Cohen (American Psychologist, 1994, 49(12), 997-1003), despite the fact that what we often want to know is given the data I have observed, what is the probability that $H_0$ is true? Anyway, there's also a nice discussion on "Why P=0.05?", by Jerry Dallal.
-
$\begingroup$ Please correct my train of thoughts: some fields might focus on, say, biochemical exposure, and hence want to use p < 0.001 as to prevent any Type I error that might lead to health hazard. Also, along this article from Am Psych, I also remember a great study in the Am J of Sociol or one of the soc sci journals that I follow. My favourite is, of course, Ziliak and McCloskey. $\endgroup$– Fr.Mar 21, 2011 at 23:19
-
1$\begingroup$ What you describe here sounds backwards. I'd be worried about Type II errors, saying something's not there when it is, with biochemical exposure. In that case I might set alpha higher, not lower. $\endgroup$– JohnMar 22, 2011 at 0:42
-
$\begingroup$ I was working under the assumption that the test would of the form: "Let's assess if pregnancy is related to HRT" (in that case, a Type I error is more serious than a Type II error, but perhaps this design is nonstandard). $\endgroup$– Fr.Mar 22, 2011 at 1:26
$\begingroup$
$\endgroup$
5
It might be rare for anyone to use a pre-specified alpha level lower than, say, 0.01, but it is not nearly as rare that people claim an implied alpha of less than 0.01 in the mistaken belief that an observed P value of less than 0.01 is the same as a Neyman-Pearson alpha of less than 0.01.
Fisher's P values are not the same as, or interchangeable with, Neyman-Pearson error rates. $P = 0.0023$ does not mean $\alpha = 0.0023$ unless one has decided to use $0.0023$ as the critical level for significance when the experiment is designed. If you would have taken $P = 0.05$ as significant then $P = 0.0023$ means that there is an $0.05$ probability of a false positive claim.
gung - Reinstate Monica
145k88 gold badges400 silver badges708 bronze badges
answered Mar 21, 2011 at 23:26
-
$\begingroup$ I understand the distinction, although I am probably making the mistake routinely. But my question is, is there any conventional usage, somewhere out there, of p < .0001 for instance? Or, to put it provocatively, is the p < .05 cult universal? $\endgroup$– Fr.Mar 21, 2011 at 23:35
-
$\begingroup$ The 'cult' of P<0.05 may be nearly universal, but it is not possible to be confident about any assertions on this point because apparent exceptions are quite likely to be the result of unknowing hybridisation of Fisher and Neyman-Pearson methods. In basic pharmacological research papers there is almost never an explicit statement regarding the use of Neyman-Pearson error rates. $\endgroup$ Mar 22, 2011 at 0:17
-
$\begingroup$ Thanks for the example. I am less and less impressed by pharmacological research, for many (not all scientific) reasons… $\endgroup$– Fr.Mar 22, 2011 at 1:28
-
2$\begingroup$ You shouldn't take my comment about basic pharmacological research as a specific criticism of that field, it is just my own particular discipline and thus the one with which I am most experienced. I am confident that you would find many areas in basic research with exactly the same shortcomings with respect to hybridized P values and error rates. $\endgroup$ Mar 22, 2011 at 3:39
-
$\begingroup$ No worries, I can easily imagine that this shortcoming travels well across fields of inquiry. $\endgroup$– Fr.Mar 23, 2011 at 16:26
$\begingroup$
$\endgroup$
1
I am not very familiar with this literature but I believe some physicists use much lower thresholds in statistical tests. However, they talk about it a little differently so social scientists might not realise the connection.
For example, if a measure is three standard deviations from the theoretical prediction, it is described as a “three sigma” deviation. Basically, this means that the parameter of interest is statistically different from the predicted value in a z test with $α = .01$. Two sigma is roughly equivalent to $α = .05$ (in fact it would be 1.96 σ). If I am not mistaken, the standard error level in physics is 5 sigma, which would be $α = 5*10^-7$ or $p < 0.0000005$.
Also, in neuroscience or epidemiology, it seems increasingly common to routinely perform some correction for multiple comparisons. The error level for each individual test can therefore be lower than $p < .01$.
-
2$\begingroup$ Genetic epidemiology routinely uses $\alpha=5\times10^{-8}$ in genomewide association studies, often regardless of the precise number of tests performed. $\endgroup$– guestNov 17, 2012 at 7:16
$\begingroup$
$\endgroup$
As noted by Gaël Laurans above statistical analyses that run into the multiple comparison problem tend to use more conservative thresholds. However, in essence they are using 0.05, but multiplied by the number of tests. It is obvious that this procedure (Bonferroni correction) can quickly lead to incredibly small p-values. That's why people in the past (in neuroscience) stopped at p<0.001. Nowadays other methods of multiple comparison corrections are used (see Markov random field theory).
