2
$\begingroup$

Statistical significance of a (variable in a) model grows with sample size. Citing Gilbert (1986):

If one uses test statistics with constant size (i.e. a constant degree of confidence), almost any simple model will be rejected given a sufficiently large data set.

...[On the other hand] any hypothesis can be maintained by testing it on a sufficiently short time series.

From a practitioner's perspective, this is troubling. Statistical significance is often used in convincing that a (variable in a) model is relevant; but once you know it grows with sample size, the argument loses much of its power.

Here is a related discussion (e.g. the answer by Mark Claesen).

Questions:
(1) How to overcome the sample-size-dependence of statistical significance?
(2) Can we adjust statistical significance to make it independent of sample size?
(3) What to use instead of statistical significance?

The goal is to find out whether a given (variable in a) model is relevant and useful.

|cite|improve this question
$\endgroup$
  • $\begingroup$ One solution is proposed in the same Gilbert's paper: For this reason, Leamer (1978) proposes that the econometrician should reduce the significance level he uses as his sample size increases. $\endgroup$ – Richard Hardy Jan 16 '15 at 15:51
4
$\begingroup$

This is why I am strong proponent of confidence intervals over statistical significance! A confidence interval tells you everything that a significance result tells you and more.

If I have a result where the 95% confidence interval for a mean is (0.1,0.2) I know that this is significantly different from 0 at the 95% level but I also know that it's still a pretty small number.

To answer your questions:

(1) How to overcome the sample-size-dependence of statistical significance?

Use confidence intervals!

(2) Can we adjust statistical significance to make it independent of sample size?

We shouldn't! Statistical significance attempts to answer the question: is X different from Y? If we have very strong evidence that X is approximately 0.1, well, it's different from 0.0.

When all you have is a hammer everything looks like a nail. If you do not care about this particular question (nail) don't use the statistical significance tool (hammer).

(3) What to use instead of statistical significance?

Use confidence intervals!

|cite|improve this answer
$\endgroup$
  • 3
    $\begingroup$ Very good points. It's possible to misuse confidence intervals too. The pitfall is that most clinicians have been trained to "just look if the CI contains 0 (or 1 for ratios)". You need to be explicit in the discussion of results and possibly even the background/methods in describing what you believe to be a clinically significant effect and whether the results were more or less consistent with that. $\endgroup$ – AdamO Jan 16 '15 at 16:20
  • 3
    $\begingroup$ I'd also add that often with large data we find ourselves making a large number of comparisons. Using appropriate multiple comparisons corrections can tame the apparent hype of results that are fueled by a stupid large sample. In addition, we may often look at subgroup analyses to further explore hypotheses (i.e. is the "weak" significant effect in the population attributable to subgroups with "strong" effect or is it consistent in all groups) in this case, graphical tools like forest plots or funnel plots make short work of presenting more refined results. $\endgroup$ – AdamO Jan 16 '15 at 16:23
3
$\begingroup$

In the standard null hypothesis significance testing (NHST) approach, statistical significance and sample size are related to two other parameters - effect size and statistical power (Cohen, 1988). These four parameters form a closed-circuit system, where, usually, one of the parameters (either sample size, or effect size) is determined via three others.

However, some researchers argue that the NHST approach is fundamentally flawed and, therefore, they advocate a different approach, based on confidence intervals, as @TrynnaDoStat mentioned. This new approach is appropriately named The New Statistics. The introduction to that approach and guidelines for switching from NHST to the new statistics can be found in this research article.

The new statistics approach is described in much more detail in the book "Understanding the new statistics: Effect sizes, confidence intervals and meta-analysis", written by the statistical reform pioneer Geoff Cumming (this book on Amazon: http://www.amazon.com/dp/041587968X).

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.

|cite|improve this answer
$\endgroup$
  • 3
    $\begingroup$ Actually, I find the name "The New Statistics" utterly inappropriate: come on, what is new about effect sizes and confidence intervals? $\endgroup$ – amoeba says Reinstate Monica Jan 16 '15 at 16:53
  • 2
    $\begingroup$ @amoeba: Come on, where is your sense of humor? Just because I forgot to use quotes, I expect people here on CV to detect sarcasm automagically :-). But, on a serious note - the "new" in the new statistics is IMHO the deviation from the rigid NHST model and wider adoption of the more flexible CI-based model. Citing the author on this issue: "the techniques are not new, but adopting them widely would be new for many researchers, as well as highly beneficial". $\endgroup$ – Aleksandr Blekh Jan 16 '15 at 17:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.