# Real-world example on significance testing with large samples

The issues that arise when one uses statistical hypothesis testing framework with large samples are dissussed on CV (see e.g. this thread). The main problem discussed there is that in the real world the null hypothesis like "there is no connection between variable X and variable Y" is almost always false (at least in domains like social studies when the research is not based on perfect controllable randomized experiment) and so it will be rejected if our sample is large enough.

I'm looking for references to empirical studies that support "null hypothesis is almost always false" statement.

I believe there is a study where some large survey with a hundreds of questions was used. For every question the authors tested the hypothesis that the answer to that question depends on the gender. They used t-test and choose the alternative hypothesis at random. As a result, they get about 45% of significant results. I read this paper some time ago but lost the reference and cannot find it now.

Are there any other studies like this? I'm looking for them to use as an illustration in statistical courses (when I teach others), to emphasis the distinctions between ”statistical significance” and ”practical significance”.

• You have to narrow down your domain. In physics you won't have an issue with large sample sizes which you describe. – Aksakal Jun 17 '16 at 13:33
• @Aksakal, I'm speaking mostly about social studies with large datasets obtained outside of perfect randomized experiment framework. I added a clarification to the question. – Ilya V. Schurov Jun 17 '16 at 22:14
• In what sense do you expect to find empirical support? Failure to reject even at sample sizes of billions or trillions doesn't indicate that the null is true. – Glen_b -Reinstate Monica Jun 26 '16 at 12:25

Page 205 of Meehl (1990) briefly describes a study of 57,000 high-school seniors in which 92% of 990 different cross-tabulations (between 45 variables; 45 choose 2 is 990) were statistically significant. Most people who've heard of this study are probably familiar with it from Cohen (1994).

Standing, Sproule, and Khouzam (1991) examined a dataset of 135 variables from 2,058 Canadian grade-school students. 4,506 of 17,936 correlation coefficients (25%) had a two-tailed $p < .001$.

In this age of big data, a similar study to Meehl's and Standing et al.'s with a very large sample size and number of variables would be nice. But we do have Kramer, Guillory, and Hancock (2014), a study of some 690,000 Facebook users that found signifcant effects that were positively microscopic, such as a reduction of positive posts in users' News Feeds decreasing the percentage of positive words in the users' own posts by $-0.1\%$ [$t(310,044) = -5.63$, $P < 0.001$, Cohen's $d = 0.02$]. What's really rich is that Kramer et al. pooh-pooh another significant effect they didn't want to find on the justification that it was tiny: "Positivity and negativity were evaluated separately given evidence that they are not simply opposite ends of the same spectrum. Indeed, negative and positive word use scarcely correlated [$r = -0.04$, $t(620,587) = -38.01$, $P < 0.001$]." (p. 8,789).

Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. doi:10.1037/0003-066X.49.12.997

Kramer, A. D., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24), 8788–8790. doi:10.1073/pnas.1320040111

Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244. doi:10.2466/pr0.1990.66.1.195

Standing, L., Sproule, R., & Khouzam, N. (1991). Empirical statistics: IV. Illustrating Meehl's sixth law of soft psychology: Everything correlates with everything. Psychological Reports, 69(1), 123–126. doi:10.2466/PR0.69.5.123-126

• Yeah! Meehl's conjecture is a keyword I was looking for! And actually the paper I described in the question is Waller, N.G., 2004. The fallacy of the null hypothesis in soft psychology. (I initially found a reference there from a paper Mindless statistics by Gerd Gigerenzer, 2004.) I will investigate other papers you mention as well. Thanks a lot, this is exactly what I was asking for! – Ilya V. Schurov Jun 26 '16 at 12:02

"null hypothesis is almost always false"

is that it underscores the sloppy way with which modern, frequentist hypothesis testing is done. If you adhere to that framework, then it is true, virtually all causal relations are in some, albeit miniscule and complex, way statistically true and can be found if one were to sample enough.

To me, it begs for a return to Fisherian testing. To remind people, Fisher never advocated a decision rule approach to significance testing, he merely said that a $p$-value should be compared to the statistical power of the analysis.

What this does is that it requires the investigator to a priori specify what they might consider a significant effect. By doing so, in the interpretation of results, it is made plainly apparent that the results are coming from an overpowered analysis. Results from overpowered analyses usually report very small effects, and the power to detect such an effect is usually very low. So while the $p$-value is very significant, the power is very small, and we call into question how "significant" these findings really are.

On the other hand, when you compare the power for the apriori effect size, overpowered analyses will have powers that are so large, and p-values that are so small they cannot be practically compared. This illustrates the intuitive discrepancy between what the researcher said they would find, and what they actually found.

For instance, suppose you have a trial to see the effect of blood pressure lowering meds. You realize, "ah! In power analyses, the investigator used a mean difference of 1.20 mmHg of blood pressure thinking that was a clinically significant effect, but in their analyses found a difference of 0.0300 mmHg with a 95% confidence interval 0.0299 - 0.0301 mmHg." And at that point you realize while these results are statistically significant, they really aren't clinically significant.

• Thanks for your explanations. Actually, I understand the difference between "statistical significance" and "practical (e.g. clinical) significance". However, I'm looking for references to empirical researches that can be used as an illustration of that kind of problems. – Ilya V. Schurov Jun 19 '16 at 15:27
• @IlyaV.Schurov What do you mean by "empirical researches"? – AdamO Jun 20 '16 at 20:49

I think you should into the American's Statistic association statement on p-values.

They do quote that big data researches should not draw any conclusions from p-values by itself.