Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 9 down vote favorite
2

I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a common standard?

share|improve this question

4 Answers

up vote 9 down vote

My opinion is that it does (and should) not depend on the field of study. For example, you may well work at a lower significance level than $p<0.001$ if, for example, you are trying to replicate a study with historical or well-established results (I can think of several studies on the Stroop effect, which had led to some controversies in the past few years). That amounts to consider a lower "threshold" within the classical Neyman-Pearson framework for testing hypothesis. However, statistical and practical (or substantive) significance is another matter.

Sidenote. The "star system" seems to have dominated scientific inquiries as early as the 70's, but see The Earth Is Round (p < .05), by J. Cohen (American Psychologist, 1994, 49(12), 997-1003), despite the fact that what we often want to know is given the data I have observed, what is the probability that $H_0$ is true? Anyway, there's also a nice discussion on "Why P=0.05?", by Jerry Dallal.

share|improve this answer
Please correct my train of thoughts: some fields might focus on, say, biochemical exposure, and hence want to use p < 0.001 as to prevent any Type I error that might lead to health hazard. Also, along this article from Am Psych, I also remember a great study in the Am J of Sociol or one of the soc sci journals that I follow. My favourite is, of course, Ziliak and McCloskey. – Fr. Mar 21 '11 at 23:19
What you describe here sounds backwards. I'd be worried about Type II errors, saying something's not there when it is, with biochemical exposure. In that case I might set alpha higher, not lower. – John Mar 22 '11 at 0:42
I was working under the assumption that the test would of the form: "Let's assess if pregnancy is related to HRT" (in that case, a Type I error is more serious than a Type II error, but perhaps this design is nonstandard). – Fr. Mar 22 '11 at 1:26
+1 for the side note and the link – Dmitrij Celov Mar 22 '11 at 9:04
up vote 5 down vote

It might be rare for anyone to use a pre-specified alpha level lower than, say, 0.01, but it is not nearly as rare that people claim an implied alpha of less than 0.01 in the mistaken belief that an observed P value of less than 0.01 is the same as a Neyman-Pearson alpha of less than 0.01.

Fisher's P values are not the same as, or interchangeable with, Neyman-Pearson error rates. $P = 0.0023$ does not mean $\alpha = 0.0023$ unless one has decided to use $0.0023$ as the critical level for significance when the experiment is designed. If you would have taken $P = 0.05$ as significant then $P = 0.0023$ means that there is an $0.05$ probability of a false positive claim.

Have a look at Hubbard et al. Confusion over Measures of Evidence (p's) versus Errors (α's) in Classical Statistical Testing. The American Statistician (2003) vol. 57 (3)

share|improve this answer
I understand the distinction, although I am probably making the mistake routinely. But my question is, is there any conventional usage, somewhere out there, of p < .0001 for instance? Or, to put it provocatively, is the p < .05 cult universal? – Fr. Mar 21 '11 at 23:35
The 'cult' of P<0.05 may be nearly universal, but it is not possible to be confident about any assertions on this point because apparent exceptions are quite likely to be the result of unknowing hybridisation of Fisher and Neyman-Pearson methods. In basic pharmacological research papers there is almost never an explicit statement regarding the use of Neyman-Pearson error rates. – Michael Lew Mar 22 '11 at 0:17
Thanks for the example. I am less and less impressed by pharmacological research, for many (not all scientific) reasons… – Fr. Mar 22 '11 at 1:28
1  
You shouldn't take my comment about basic pharmacological research as a specific criticism of that field, it is just my own particular discipline and thus the one with which I am most experienced. I am confident that you would find many areas in basic research with exactly the same shortcomings with respect to hybridized P values and error rates. – Michael Lew Mar 22 '11 at 3:39
No worries, I can easily imagine that this shortcoming travels well across fields of inquiry. – Fr. Mar 23 '11 at 16:26
up vote 2 down vote

I am not very familiar with this literature but I believe some physicists use much lower thresholds in statistical tests but they talk about it a little differently. For example, if a measure is three standard deviations from the theoretical prediction, it is described as a “three sigma” deviation. Basically, this means that the parameter of interest is statistically different from the predicted value in a z test with α = .01. Two sigma is roughly equivalent to α = .05 (in fact it would be 1.96 σ). If I am not mistaken, the standard error level in physics is 5 sigma, which would be α = 5*10^-7

Also, in neuroscience or epidemiology, it seems increasingly common to routinely perform some correction for multiple comparisons. The error level for each individual test can therefore be lower than p < .01

share|improve this answer
1  
Genetic epidemiology routinely uses $\alpha=5\times10^{-8}$ in genomewide association studies, often regardless of the precise number of tests performed. – guest Nov 17 '12 at 7:16
up vote 1 down vote

As noted by Gaël Laurans above statistical analyses that run into the multiple comparison problem tend to use more conservative thresholds. However, in essence they are using 0.05, but multiplied by the number of tests. It is obvious that this procedure (Bonferroni correction) can quickly lead to incredibly small p-values. That's why people in the past (in neuroscience) stopped at p<0.001. Nowadays other methods of multiple comparison corrections are used (see Markov random field theory).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.