4
$\begingroup$

This question is inspired by the discussion here. The background is that often point null hypotheses like $\mu=0$ are tested, $\mu$ being some effect of interest in some suitably parametrised model, and people argue that in reality $\mu=0$ precisely will often not occur, but in terms of interpretation $|\mu|\le\epsilon$ for some sufficiently small $\epsilon$ is basically the same as saying that "there is no effect" (meaning then "any existing effect is so small as to not be of practical relevance). A well known issue with hypothesis tests is that in such a case, when testing $H_0:\ \mu=0$ and indeed $\mu=\frac{\epsilon}{2}$, say, a large enough data set will reject $H_0$, as the test has then enough information to see that $\mu\neq 0$ even though very small.

Equivalence tests test a null hypothesis of the kind $|\mu|\le\epsilon$ with pre-specified $\epsilon$, and it has been argued that because of the above mentioned defect of standard tests, people should use and teach equivalence tests instead (which of course require to specify $\epsilon$).

Another approach would be to just on top of the standard test compute a confidence interval and see whether something as small as $\epsilon$ is in it. I think people should generally not only be interested in rejecting (or not rejecting) a null hypothesis, but also in effect sizes, so computing a confidence interval and interpreting it seems to be generally good advice. Given that this is so, I wonder what equivalence testing actually adds. Are there any reasons to run an equivalence test given that we can well address the same issue using a confidence interval (the confidence interval has the additional advantage that $\epsilon$ doesn't need to be specified, and one can rather look at the CI and see whether something is in there that seems "too small to be meaningful" a posteriori)? And even if situations can be constructed in which the equivalence test adds something worthwhile, could these situations be so exotic and exceptional that it basically isn't worth the hassle to teach and advertise them big time?

PS: Addition prompted by @FrankHarrell's answer: Certain Bayesians think that neither of these should be taught or used, which within the Bayesian paradigm is fair enough. I disagree (though I'm not against Bayes in general), but the question is obviously set within a frequentist framework (not necessarily claiming that this should dominate; even if you favour Bayes in most situations you may use frequentist approaches from time to time), and is not intended to have a Bayesians vs. frequentist discussion here.

$\endgroup$
1
  • $\begingroup$ I agree that there is very little need for equivalence tests for that purpose. Not only do we have the CI, we should also make use of the distinction between statistical significance and practical significance when reporting. For example, "the estimated effect size is both statistically and medically significant", or, "the estimated effect size is statistically significant but not economically significant". $\endgroup$ Commented Nov 8, 2022 at 20:37

2 Answers 2

1
$\begingroup$

As a Bayesian, I am puzzled why the frequentist way of thinking is still the default. With Bayesian posterior inference you don't need special cases or different study designs to answer questions about superiority vs. questions about similarity/"equivalence". You just compute more posterior probabilities, and you capitalize on directionality. A single study can provide evidence for superiority, similarity, and harm, and can also address multiple response variables. Examples: P(effect > 0), P(effect > 5%), P(effect < -2%), P(effect on response 1 > 2% and effect on response 2 > 3%).

In general computing a probability that the effect is in an investigator-specified interval is more useful than finding an uncertainty interval attached to a given investigator-specified coverage probability.

$\endgroup$
2
  • $\begingroup$ Fair enough, one could discuss for a long time how much of a problem it is that one has to specify a prior and how to do that, but the question is about equivalence tests vs. confidence intervals, so it is set within the frequentist paradigm, and that some may generally prefer a Bayesian approach for good or not so good reasons is not the point here. $\endgroup$ Commented Nov 8, 2022 at 11:42
  • 2
    $\begingroup$ It is IMHO a good reason, if the frequentist paradigm requires so many one-off solutions and doesn't even intend to provide direct evidence in support of various assertions. $\endgroup$ Commented Nov 8, 2022 at 11:52
1
$\begingroup$

Thinking about it, I could see two advantages:

  • In pre-registered studies, one may then choose $\epsilon$ a priori, based on certain arguments, such that $|\mu|<\epsilon$ would may be deemed uninteresting but anything above would be perceived as most likely relevant/meaningful. This makes it harder to move the goalpost if the effect does not have the expected strength.
  • The value of $\epsilon$ is also a basis for choosing a sample size that gives the test the proper amount of statistical power. So, specifying $\epsilon$ beforehand may help planning the study anyway. This may help counter the argument according to which the null will always be rejected provided the sample size is large enough (by making it just large enough to find interesting effects, but not much larger than that).

This makes null hypothesis testing a little closer to ideal Popperian falsification, by committing to stricter requirements for rejecting the null prior to testing, and by making the null hypothesis a more challenging contender to the alternative.

Now is your question that, even if one accepts to commit to a certain choice of $\epsilon$ prior to the study, then CI remains more informative and should be preferred? If this is about being more informative then, I agree with @Frank Harrell's Bayesian take, that the posterior distribution of the effect size would be more interesting anyway.

$\endgroup$
2
  • 1
    $\begingroup$ These are reasons for choosing $\epsilon$ beforehand and I agree that it is good to think about meaningful effect sizes before data. However given that an $\epsilon$ has been chosen, for everything mentioned here I suspect one could still look at the CI and doesn't need an equivalence test? $\endgroup$ Commented Nov 8, 2022 at 17:55
  • $\begingroup$ Alright, I had answered more specifically about CIs not requiring to choose $\epsilon$ prior, so I edited my answer to make it slightly broader in scope. $\endgroup$ Commented Nov 9, 2022 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.