2
$\begingroup$

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this quite often and I need references as to why this is true. Any help is appreciated.

$\endgroup$
2
1
$\begingroup$

The reason is that as sample size increases your ability to detect deviations from the null (no effect) increases [standard errors of the estimate shrink]. Therefore, at sufficiently large sample sizes, even tiny effects may become statistically significant (i.e. $p < \alpha$). It is always (or at least almost nearly always) an appropriate thing to do to look at the effect size of your statistic. The effect size will quantify how large your effect is, hopefully in terms you care about (e.g. proportion of variance accounted for or magnitude of average difference scaled by error in measurement). If you know how to interpret your effect size, then you will also know how to judge whether this is a tiny effect of no theoretical interest, or an effect large enough to be worthy of consideration.

$\endgroup$
6
  • $\begingroup$ Hi @rpierce by "standard errors of the estimate shrink", you mean standard error of the mean SEM? $\endgroup$ Jun 18 '14 at 20:14
  • 1
    $\begingroup$ @LadislavNado if the inference is on the mean then yes. But odds ratios, proportions, rates, etc. all have associated standard errors. As with mean estimation, test statistics for these tests will reach "highly improbably domains" according to the null hypothesis as sample size increases. $\endgroup$
    – AdamO
    Jun 18 '14 at 20:17
  • 1
    $\begingroup$ @rpierce As a note, there is inconsistent usage of the terms "effect sizes" in the literature. Some take "effect size" to mean "association measure" which does not vary with sample size, and I think that was your intended usage. Others use effect size to mean test statistics which do vary with sample size. $\endgroup$
    – AdamO
    Jun 18 '14 at 20:18
  • $\begingroup$ Oh no...who uses effect size to refer to test statistics? How could that possibly be valid usage? $\endgroup$ Jun 18 '14 at 20:24
  • $\begingroup$ @AdamO yes that was my intended usage. I didn't know there was terminology confusion there. Thanks for letting me know, egads. $\endgroup$ Jun 18 '14 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.