I am working with a data set having N around 200,000. In regressions, I am seeing very small significance values << 0.001 associated with very small effect sizes, e.g. r=0.028. What I'd like to know is, is there a principled way of deciding an appropriate significance threshold in relation to the sample size? Are there any other important considerations about interpreting effect size with such a large sample?
$\begingroup$
$\endgroup$
7
-
10$\begingroup$ This is an issue of practical vs. statistical significance. If the slope is truly different from 0, even by a miniscule amount e.g. .00000000000001), a large enough sample will yield a very tiny $p$-value, despite the result having no practical significance. You'd do better interpreting the point estimate rather than the $p$-value when you have such a large sample size. $\endgroup$– MacroFeb 3, 2012 at 19:34
-
$\begingroup$ @Macro sorry can you clarify what you mean by point estimate here? $\endgroup$– ted.straussFeb 3, 2012 at 19:43
-
3$\begingroup$ Adding to Macro's comment above, in this situation I look for "practical" or "clinical" significance in the findings. For what you are doing, is the effect large enough for you to care? $\endgroup$– MichelleFeb 3, 2012 at 19:54
-
1$\begingroup$ The point estimate is the observed regression slope estimate. $\endgroup$– MacroFeb 3, 2012 at 20:22
-
2$\begingroup$ What @Macro and I are both saying is that you need to decide whether the clinical effect (point estimates, slopes) is important. Your threshold is on the basis of deciding "yes, this is an important clinical effect" rather than "a significant p-value" because most (all?) of your p-values are significant. $\endgroup$– MichelleFeb 3, 2012 at 21:08
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
5
In The insignificance of significance testing, Johnson (1999) noted that p-values are arbitrary, in that you can make them as small as you wish by gathering enough data, assuming the null hypothesis is false, which it almost always is. In the real world, there are unlikely to be semi-partial correlations that are exactly zero, which is the null hypothesis in testing significance of a regression coefficient. P-value significance cutoffs are even more arbitrary. The value of .05 as the cutoff between significance and nonsignificance is used by convention, not on principle. So the answer to your first question is no, there is no principled way to decide on an appropriate significance threshold.
So what can you do, given your large data set? It depends on your reason(s) for exploring the statistical significance of your regression coefficients. Are you trying to model a complex multi-factorial system and develop a useful theory that reasonably fits or predicts reality? Then maybe you could think about developing a more elaborate model and taking a modeling perspective on it, as described in Rodgers (2010), The Epistemology of Mathematical And Statistical Modeling. One advantage of having a lot of data is being able to explore very rich models, ones with multiple levels and interesting interactions (assuming you have the variables to do so).
If, on the other hand, you want to make some judgement as to whether to treat a particular coefficient as statistically significant or not, you might want to take Good's (1982) suggestion as summarized in Woolley (2003): Calculate the q-value as $p\cdot\sqrt{(n/100)}$ which standardizes p-values to a sample size of 100. A p-value of exactly .001 converts to a p-value of .045 -- statistically significant still.
So if it's significant using some arbitrary threshold or another, what of it? If this is an observational study you have a lot more work to justify that it's actually meaningful in the way you think and not just a spurious relationship that shows up because you have misspecified your model. Note that a small effect is not so clinically interesting if it represents pre-existing differences across people selecting into different levels of treatment rather than a treatment effect.
You do need to consider whether the relationship you're seeing is practically significant, as commenters have noted. Converting the figures you quote from $r$ to $r^2$ for variance explained ($r$ is correlation, square it to get variance explained) gives just 3 and 6% variance explained, respectively, which doesn't seem like much.
-
$\begingroup$ @rolando2 thanks for the edit, always getting confused between large/small p-values! I think if it's off to right of distribution it's large, but p-value is small. $\endgroup$– Anne Z.Feb 4, 2012 at 0:27
-
2$\begingroup$ (+1) This is an important fact that many practitioners do not think carefully about: "p-values are arbitrary, in that you can make them as small as you wish by gathering enough data, assuming the null hypothesis is false, which it almost always is." $\endgroup$– MacroFeb 4, 2012 at 3:53
-
$\begingroup$ Thank you! The points in your penultimate paragraph are well taken. I am reading the Woolley article and noticed that your q-value formula is off. It should be p* not p/ - I tried to change it here but edits must be >6 characters. $\endgroup$ Feb 4, 2012 at 17:07
-
$\begingroup$ @ted.strauss I'm glad it's helpful. Sometimes I feel discouraged by the limitations of the tools like p-values that we have to work with. Thanks for noting the mistake in the formula, I've fixed it. $\endgroup$– Anne Z.Feb 4, 2012 at 18:08
-
$\begingroup$ Thanks for the wonderful answer. But I am not able to access the paper Woolley 2003 using the link provided above. $\endgroup$– KarthikSMar 31, 2016 at 4:53
$\begingroup$
$\endgroup$
4
I guess an easy way to check would be randomly sampling a similarly large number from what you know is one distribution twice and comparing the two results. If you do that several times and observe similar p-values, it would suggest that there's no real effect. If on the other hand you don't, then there probably is.
-
8$\begingroup$ I think you're suggesting doing simulations under the null hypothesis of no true difference with a large sample size and looking at the $p$-values. I can tell you without doing the simulations that $<.001$ proportion of the resulting $p$-values will be as small as the one the original poster observed. This is true for any sample size. This is the definition of a $p$-value. $\endgroup$– MacroFeb 3, 2012 at 20:20
-
1$\begingroup$ In fact, the $p$-values that will come out of the process you described will have a ${\rm Uniform}(0,1)$ distribution. $\endgroup$– MacroFeb 3, 2012 at 20:22
-
2$\begingroup$ In relation to the last comment by @Macro, here is a sketch of the proof that, under the null hypothesis $H_0$, the $p$-value has $U[0,1]$ distribution. Given a test statistic $T=T(X)$, if we observe $t=t(x)$, the $p$-value is defined as $p(t)=\mathbb{P}(T\geq t\mid H_0)$. Suppose that under $H_0$ the distribution function of $T$ is $G_0$, with $G_0$ continuous and nondecreasing, so that it has inverse $G_0^{-1}$. Then, we have $p(t)=1-G_0(t)$, and, for $u\in[0,1]$ $\endgroup$– ZenFeb 27, 2012 at 21:56
-
2$\begingroup$ (continuation of Zen's comment): $$ \mathbb{P}(p(T)\leq u) = \mathbb{P}(1-G_0(T)\leq u) = \mathbb{P}(G_0(T)\geq 1-u) = \mathbb{P}(T\geq G_0^{-1}(1-u)) = 1-G_0(G_0^{-1}(1-u))=u \, . $$ Hence, we conclude that $p(T)\mid H_0\sim U[0,1]$. $\endgroup$– whuber ♦Feb 27, 2012 at 22:19