# Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?

I was just reading this article on the Bayes factor for a completely unrelated problem when I stumbled upon this passage

Hypothesis testing with Bayes factors is more robust than frequentist hypothesis testing, since the Bayesian form avoids model selection bias, evaluates evidence in favor the null hypothesis, includes model uncertainty, and allows non-nested models to be compared (though of course the model must have the same dependent variable). Also, frequentist significance tests become biased in favor of rejecting the null hypothesis with sufficiently large sample size. [emphasis added]

I've seen this claim before in Karl Friston's 2012 paper in NeuroImage, where he calls it the fallacy of classical inference.

I've had a bit of trouble finding a truly pedagogical account of why this should be true. Specifically, I'm wondering:

1. why this occurs
2. how to guard against it
3. failing that, how to detect it
-
It's somewhat debatable because it's untrue when the null is literally, exactly true, but since that is so rarely the case (due to all sorts of complexities like spurious correlations), it's probably true of most practical applications. Hypothetically speaking, one could detect the weakest of spurious correlations (e.g., r = .001) due to a chain of mediators hundreds of variables long despite a similar number of uncontrolled moderators if the sample was colossal enough. Arguably, that relationship actually exists though, so whether that's really "bias" is still somewhat debatable IMO... –  Nick Stauner Jul 22 '14 at 20:34
@NickStauner, Ah that actually makes a lot of sense! Thanks for the intuitive explanation! –  blz Jul 22 '14 at 20:36
Tal Yarkoni wrote a very enlightening critique of Friston's article: talyarkoni.org/blog/2012/04/25/… –  jona Jul 22 '14 at 22:33
@jona, Seems like I'm running into the whole cogsci crowd over here =) Thanks for the reference, this does indeed look like good reading! –  blz Jul 22 '14 at 22:49
Given the assumptions hold, that statement seems to be strictly false as it stands, but it is getting at a real issue (that with sufficiently large samples, a NHST will become almost certain to reject a false null, no matter how tiny the effect). When people find that a problem, it usually indicates that hypothesis testing isn't what they need. The same basic issue (though framed in terms of CIs rather than hypothesis tests) is discussed in this answer –  Glen_b Jul 22 '14 at 22:58

Answer to question 1: This occurs because the $p$-value becomes arbitrarily small as the sample size increases in frequentist tests for difference (i.e. tests with a null hypothesis of no difference/some form of equality). The $p$-value becomes arbitrarily small because the error of frequentist test statistics generally decreases with sample size, with the upshot that all differences are significant to an arbitrary level with a large enough sample size.

Answer to question 2: Within a frequentist hypothesis testing framework, one can guard against this by not making inference solely about detecting difference. For example, one can combine inferences about difference and equivalence so that one is not favoring (or conflating!) the burden of proof on evidence of effect versus evidence of absence of effect. Evidence of absence of an effect comes from, for example:

1. two one-sided tests for equivalence (TOST),
2. uniformly most powerful tests for equivalence, and
3. the confidence interval approach to equivalence (i.e. if the $1-2\alpha$%CI of the test statistic is within the a priori-defined range of equivalence/relevance, then one concludes equivalence at the $\alpha$ level of significance).

What these approaches all share is an a priori decision about what effect size constitutes a relevant difference and a null hypothesis framed in terms of a difference at least as large as what is considered relevant.

Combined inference from tests for difference and tests for equivalence thus protects against the bias you describe when sample sizes are large in this way (two-by-two table showing the four possibilities resulting from combined tests for difference—positivist null hypothesis, H$_{0}^{+}$—and equivalence—negativist null hypothesis, H$_{0}^{-}$):

Notice the upper left quadrant: an overpowered test is one where yes you reject the null hypothesis of no difference, but you also reject the null hypothesis of relevant difference, so yes there's a difference, but you have a priori decided you do not care about it because it is too small.

These combined tests are called "relevance tests" and yet only little studied. However, a (conservative) relevance decision can be found if one rejects the Null hypothesis iff the usual $1-\alpha$-confidence interval is disjount from the relevance region. So, @Alexis, in case of relevance tests, you take $\alpha$, in case of equivalence testing, you take $2\alpha$. –  Horst Grünbusch Jul 27 '14 at 14:22
Belief that frequentist test are not biased towards rejecting the null hypothesis as sample size grows is based on assuming that $0$ is meaningfully and substantively different than $0 + \text{really frickin' tiny}$. –  Alexis May 8 at 17:18