Real-world example on significance testing with large samples The issues that arise when one uses statistical hypothesis testing framework with large samples are dissussed on CV (see e.g. this thread). The main problem discussed there is that in the real world the null hypothesis like "there is no connection between variable X and variable Y" is almost always false (at least in domains like social studies when the research is not based on perfect controllable randomized experiment) and so it will be rejected if our sample is large enough.
I'm looking for references to empirical studies that support "null hypothesis is almost always false" statement.
I believe there is a study where some large survey with a hundreds of questions was used. For every question the authors tested the hypothesis that the answer to that question depends on the gender. They used t-test and choose the alternative hypothesis at random. As a result, they get about 45% of significant results. I read this paper some time ago but lost the reference and cannot find it now.
Are there any other studies like this? I'm looking for them to use as an illustration in statistical courses (when I teach others), to emphasis the distinctions between ”statistical significance” and ”practical significance”.
 A: The thing that I find so teethgrating about this expression: 

"null hypothesis is almost always false"

is that it underscores the sloppy way with which modern, frequentist hypothesis testing is done. If you adhere to that framework, then it is true, virtually all causal relations are in some, albeit miniscule and complex, way statistically true and can be found if one were to sample enough. 
To me, it begs for a return to Fisherian testing. To remind people, Fisher never advocated a decision rule approach to significance testing, he merely said that a $p$-value should be compared to the statistical power of the analysis.
What this does is that it requires the investigator to a priori specify what they might consider a significant effect. By doing so, in the interpretation of results, it is made plainly apparent that the results are coming from an overpowered analysis. Results from overpowered analyses usually report very small effects, and the power to detect such an effect is usually very low. So while the $p$-value is very significant, the power is very small, and we call into question how "significant" these findings really are.
On the other hand, when you compare the power for the apriori effect size, overpowered analyses will have powers that are so large, and p-values that are so small they cannot be practically compared. This illustrates the intuitive discrepancy between what the researcher said they would find, and what they actually found.
For instance, suppose you have a trial to see the effect of blood pressure lowering meds. You realize, "ah! In power analyses, the investigator used a mean difference of 1.20 mmHg of blood pressure thinking that was a clinically significant effect, but in their analyses found a difference of 0.0300 mmHg with a 95% confidence interval 0.0299 - 0.0301 mmHg." And at that point you realize while these results are statistically significant, they really aren't clinically significant.
A: I think you should into the American's Statistic association statement on p-values.
They do quote that big data researches should not draw any conclusions from p-values by itself.
