48
$\begingroup$

I was just reading this article on the Bayes factor for a completely unrelated problem when I stumbled upon this passage

Hypothesis testing with Bayes factors is more robust than frequentist hypothesis testing, since the Bayesian form avoids model selection bias, evaluates evidence in favor the null hypothesis, includes model uncertainty, and allows non-nested models to be compared (though of course the model must have the same dependent variable). Also, frequentist significance tests become biased in favor of rejecting the null hypothesis with sufficiently large sample size. [emphasis added]

I've seen this claim before in Karl Friston's 2012 paper in NeuroImage, where he calls it the fallacy of classical inference.

I've had a bit of trouble finding a truly pedagogical account of why this should be true. Specifically, I'm wondering:

  1. why this occurs
  2. how to guard against it
  3. failing that, how to detect it
| cite | improve this question | |
$\endgroup$
  • 7
    $\begingroup$ It's somewhat debatable because it's untrue when the null is literally, exactly true, but since that is so rarely the case (due to all sorts of complexities like spurious correlations), it's probably true of most practical applications. Hypothetically speaking, one could detect the weakest of spurious correlations (e.g., r = .001) due to a chain of mediators hundreds of variables long despite a similar number of uncontrolled moderators if the sample was colossal enough. Arguably, that relationship actually exists though, so whether that's really "bias" is still somewhat debatable IMO... $\endgroup$ – Nick Stauner Jul 22 '14 at 20:34
  • $\begingroup$ @NickStauner, Ah that actually makes a lot of sense! Thanks for the intuitive explanation! $\endgroup$ – blz Jul 22 '14 at 20:36
  • 3
    $\begingroup$ Tal Yarkoni wrote a very enlightening critique of Friston's article: talyarkoni.org/blog/2012/04/25/… $\endgroup$ – jona Jul 22 '14 at 22:33
  • $\begingroup$ @jona, Seems like I'm running into the whole cogsci crowd over here =) Thanks for the reference, this does indeed look like good reading! $\endgroup$ – blz Jul 22 '14 at 22:49
  • 8
    $\begingroup$ Given the assumptions hold, that statement seems to be strictly false as it stands, but it is getting at a real issue (that with sufficiently large samples, a NHST will become almost certain to reject a false null, no matter how tiny the effect). When people find that a problem, it usually indicates that hypothesis testing isn't what they need. The same basic issue (though framed in terms of CIs rather than hypothesis tests) is discussed in this answer $\endgroup$ – Glen_b Jul 22 '14 at 22:58
47
+150
$\begingroup$

Answer to question 1: This occurs because the $p$-value becomes arbitrarily small as the sample size increases in frequentist tests for difference (i.e. tests with a null hypothesis of no difference/some form of equality) when a true difference exactly equal to zero, as opposed to arbitraily close to zero, is not realistic (see Nick Stauner's comment to the OP). The $p$-value becomes arbitrarily small because the error of frequentist test statistics generally decreases with sample size, with the upshot that all differences are significant to an arbitrary level with a large enough sample size. Cosma Shalizi has written eruditely about this.

Answer to question 2: Within a frequentist hypothesis testing framework, one can guard against this by not making inference solely about detecting difference. For example, one can combine inferences about difference and equivalence so that one is not favoring (or conflating!) the burden of proof on evidence of effect versus evidence of absence of effect. Evidence of absence of an effect comes from, for example:

  1. two one-sided tests for equivalence (TOST),
  2. uniformly most powerful tests for equivalence, and
  3. the confidence interval approach to equivalence (i.e. if the $1-2\alpha$%CI of the test statistic is within the a priori-defined range of equivalence/relevance, then one concludes equivalence at the $\alpha$ level of significance).

What these approaches all share is an a priori decision about what effect size constitutes a relevant difference and a null hypothesis framed in terms of a difference at least as large as what is considered relevant.

Combined inference from tests for difference and tests for equivalence thus protects against the bias you describe when sample sizes are large in this way (two-by-two table showing the four possibilities resulting from combined tests for difference—positivist null hypothesis, H$_{0}^{+}$—and equivalence—negativist null hypothesis, H$_{0}^{-}$):

Four possibilities from combined tests for difference and tests for equivalence

Notice the upper left quadrant: an overpowered test is one where yes you reject the null hypothesis of no difference, but you also reject the null hypothesis of relevant difference, so yes there's a difference, but you have a priori decided you do not care about it because it is too small.

Answer to question 3: See answer to 2.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Answers like this one are why I keep coming here. Thank you! $\endgroup$ – blz Jul 22 '14 at 21:14
  • 2
    $\begingroup$ These combined tests are called "relevance tests" and yet only little studied. However, a (conservative) relevance decision can be found if one rejects the Null hypothesis iff the usual $1-\alpha$-confidence interval is disjount from the relevance region. So, @Alexis, in case of relevance tests, you take $\alpha$, in case of equivalence testing, you take $2\alpha$. $\endgroup$ – Horst Grünbusch Jul 27 '14 at 14:22
  • $\begingroup$ To supplement the answer to Question 1, a relevant blog post from Cosma Shalizi $\endgroup$ – user5594 Jan 21 '16 at 18:33
  • 4
    $\begingroup$ I'm a bit surprised that everyone finds this question so helpful although the "Answer to question 1" is actually much more appropriately answered by Michael Lew - Alexis, as it seems nearly clear that this will stay up, maybe you could correct your answer to say that, mathematically speaking, hypothesis tests are in fact NOT BIASED by large sample size, according to the normal definition of bias (the other way around actually, small sample size can be a problem)! $\endgroup$ – Florian Hartig Jan 21 '16 at 18:43
  • 5
    $\begingroup$ I understand the problem, and I agree with the assessment - it's uninformative or misleading to do a hypothesis test when !H0 is infinitely likely in the first place, and you have power close to 1. But that doesn't make the test biased, unless your definition of bias is that a method gives the right result to a question that you think should not be asked. $\endgroup$ – Florian Hartig Jan 23 '16 at 10:00
24
$\begingroup$

Frequentist tests with large samples DO NOT exhibit bias towards rejecting the null hypothesis if the null hypothesis is true. If the assumptions of the test are valid and the null hypothesis is true then there is no more risk of a large sample leading to rejection of the null hypothesis than a small sample. If the null is not true then we surely would be pleased to reject it, so the fact that a large sample will more frequently reject a false null than a small sample is not 'bias' but appropriate behaviour.

The fear of 'overpowered experiments' is based on assuming that it is not a good thing to reject the null hypothesis when it is nearly true. But if it is only nearly true then it is actually false! Reject away, but do not fail to notice (and clearly report) the effect size observed. It may be trivially small and therefore not worthy of serious consideration, but a decision on that issue has to be made after consideration of information from outside the hypothesis test.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Belief that frequentist test are not biased towards rejecting the null hypothesis as sample size grows is based on assuming that $0$ is meaningfully and substantively different than $0 + \text{really frickin' tiny}$. $\endgroup$ – Alexis May 8 '15 at 17:18
  • 6
    $\begingroup$ @Alexis Read the second paragraph again. I absolutely agree that really frickin' tiny is not substantively important, but it is also not logically zero. $\endgroup$ – Michael Lew May 8 '15 at 21:00
  • 6
    $\begingroup$ Sorry for a comment that is worthless to the public, but @MichaelLew, I really liked your answer. The first sentence is quite important and I do not think it was elucidated efficiently in Alexis' answer (which is also nice, of course). $\endgroup$ – Richard Hardy Jan 21 '16 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.