I am working with a data set having N around 200,000. In regressions, I am seeing very small significance values << 0.001 associated with very small effect sizes, e.g. r=0.028. What I'd like to know is, is there a principled way of deciding an appropriate significance threshold in relation to the sample size? Are there any other important considerations about interpreting effect size with such a large sample?

  • 10
    $\begingroup$ This is an issue of practical vs. statistical significance. If the slope is truly different from 0, even by a miniscule amount e.g. .00000000000001), a large enough sample will yield a very tiny $p$-value, despite the result having no practical significance. You'd do better interpreting the point estimate rather than the $p$-value when you have such a large sample size. $\endgroup$ – Macro Feb 3 '12 at 19:34
  • $\begingroup$ @Macro sorry can you clarify what you mean by point estimate here? $\endgroup$ – ted.strauss Feb 3 '12 at 19:43
  • 3
    $\begingroup$ Adding to Macro's comment above, in this situation I look for "practical" or "clinical" significance in the findings. For what you are doing, is the effect large enough for you to care? $\endgroup$ – Michelle Feb 3 '12 at 19:54
  • 1
    $\begingroup$ The point estimate is the observed regression slope estimate. $\endgroup$ – Macro Feb 3 '12 at 20:22
  • 2
    $\begingroup$ What @Macro and I are both saying is that you need to decide whether the clinical effect (point estimates, slopes) is important. Your threshold is on the basis of deciding "yes, this is an important clinical effect" rather than "a significant p-value" because most (all?) of your p-values are significant. $\endgroup$ – Michelle Feb 3 '12 at 21:08

In The insignificance of significance testing, Johnson (1999) noted that p-values are arbitrary, in that you can make them as small as you wish by gathering enough data, assuming the null hypothesis is false, which it almost always is. In the real world, there are unlikely to be semi-partial correlations that are exactly zero, which is the null hypothesis in testing significance of a regression coefficient. P-value significance cutoffs are even more arbitrary. The value of .05 as the cutoff between significance and nonsignificance is used by convention, not on principle. So the answer to your first question is no, there is no principled way to decide on an appropriate significance threshold.

So what can you do, given your large data set? It depends on your reason(s) for exploring the statistical significance of your regression coefficients. Are you trying to model a complex multi-factorial system and develop a useful theory that reasonably fits or predicts reality? Then maybe you could think about developing a more elaborate model and taking a modeling perspective on it, as described in Rodgers (2010), The Epistemology of Mathematical And Statistical Modeling. One advantage of having a lot of data is being able to explore very rich models, ones with multiple levels and interesting interactions (assuming you have the variables to do so).

If, on the other hand, you want to make some judgement as to whether to treat a particular coefficient as statistically significant or not, you might want to take Good's (1982) suggestion as summarized in Woolley (2003): Calculate the q-value as $p\cdot\sqrt{(n/100)}$ which standardizes p-values to a sample size of 100. A p-value of exactly .001 converts to a p-value of .045 -- statistically significant still.

So if it's significant using some arbitrary threshold or another, what of it? If this is an observational study you have a lot more work to justify that it's actually meaningful in the way you think and not just a spurious relationship that shows up because you have misspecified your model. Note that a small effect is not so clinically interesting if it represents pre-existing differences across people selecting into different levels of treatment rather than a treatment effect.

You do need to consider whether the relationship you're seeing is practically significant, as commenters have noted. Converting the figures you quote from $r$ to $r^2$ for variance explained ($r$ is correlation, square it to get variance explained) gives just 3 and 6% variance explained, respectively, which doesn't seem like much.

  • $\begingroup$ @rolando2 thanks for the edit, always getting confused between large/small p-values! I think if it's off to right of distribution it's large, but p-value is small. $\endgroup$ – Anne Z. Feb 4 '12 at 0:27
  • 2
    $\begingroup$ (+1) This is an important fact that many practitioners do not think carefully about: "p-values are arbitrary, in that you can make them as small as you wish by gathering enough data, assuming the null hypothesis is false, which it almost always is." $\endgroup$ – Macro Feb 4 '12 at 3:53
  • $\begingroup$ Thank you! The points in your penultimate paragraph are well taken. I am reading the Woolley article and noticed that your q-value formula is off. It should be p* not p/ - I tried to change it here but edits must be >6 characters. $\endgroup$ – ted.strauss Feb 4 '12 at 17:07
  • $\begingroup$ @ted.strauss I'm glad it's helpful. Sometimes I feel discouraged by the limitations of the tools like p-values that we have to work with. Thanks for noting the mistake in the formula, I've fixed it. $\endgroup$ – Anne Z. Feb 4 '12 at 18:08
  • $\begingroup$ Thanks for the wonderful answer. But I am not able to access the paper Woolley 2003 using the link provided above. $\endgroup$ – KarthikS Mar 31 '16 at 4:53

I guess an easy way to check would be randomly sampling a similarly large number from what you know is one distribution twice and comparing the two results. If you do that several times and observe similar p-values, it would suggest that there's no real effect. If on the other hand you don't, then there probably is.

  • 7
    $\begingroup$ I think you're suggesting doing simulations under the null hypothesis of no true difference with a large sample size and looking at the $p$-values. I can tell you without doing the simulations that $<.001$ proportion of the resulting $p$-values will be as small as the one the original poster observed. This is true for any sample size. This is the definition of a $p$-value. $\endgroup$ – Macro Feb 3 '12 at 20:20
  • 1
    $\begingroup$ In fact, the $p$-values that will come out of the process you described will have a ${\rm Uniform}(0,1)$ distribution. $\endgroup$ – Macro Feb 3 '12 at 20:22
  • 1
    $\begingroup$ In relation to the last comment by @Macro, here is a sketch of the proof that, under the null hypothesis $H_0$, the $p$-value has $U[0,1]$ distribution. Given a test statistic $T=T(X)$, if we observe $t=t(x)$, the $p$-value is defined as $p(t)=\mathbb{P}(T\geq t\mid H_0)$. Suppose that under $H_0$ the distribution function of $T$ is $G_0$, with $G_0$ continuous and nondecreasing, so that it has inverse $G_0^{-1}$. Then, we have $p(t)=1-G_0(t)$, and, for $u\in[0,1]$ $\endgroup$ – Zen Feb 27 '12 at 21:56
  • 1
    $\begingroup$ (continuation of Zen's comment): $$ \mathbb{P}(p(T)\leq u) = \mathbb{P}(1-G_0(T)\leq u) = \mathbb{P}(G_0(T)\geq 1-u) = \mathbb{P}(T\geq G_0^{-1}(1-u)) = 1-G_0(G_0^{-1}(1-u))=u \, . $$ Hence, we conclude that $p(T)\mid H_0\sim U[0,1]$. $\endgroup$ – whuber Feb 27 '12 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.