Can we accept the null in noninferiority tests?

Question

In a usual t-test of means, using the usual hypothesis testing methods, we either reject the null or fail to reject the null but we never accept the null. One reason for this is that if we got more evidence, the same effect size would become significant.

But what happens in a noninferiority test?

That is:

$$H_0: \mu_1 - \mu_0 \le x$$

vs.

$$H_1: \mu_1 - \mu_0 > x$$

where $x$ is some amount that we regard as essentially the same. So, if we reject the null we say that $\mu_1$ is greater than $\mu_0$ by at least $x$. We fail to reject the null if there is insufficient evidence.

If the effect size is $x$ or greater, then this is analogous to the regular t-test. But what if the effect size is less than $x$ in the sample we have? Then, if we increased the sample size and kept the same effect, it would stay nonsignificant. Can we, therefore, accept the null in this case?

Answer 1

Your logic applies in exactly the same way to the good old one-sided tests (i.e. with $x=0$) that may be more familiar to the readers. For concreteness, imagine we are testing the null $H_0:\mu\le0$ against the alternative that $\mu$ is positive. Then if true $\mu$ is negative, increasing sample size will not yield a significant result, i.e., to use your words, it is not true that "if we got more evidence, the same effect size would become significant".

If we test $H_0:\mu\le 0$, we can have three possible outcomes:

1. First, $(1-\alpha)\cdot100\%$ confidence interval can be entirely above zero; then we reject the null and accept the alternative (that $\mu$ is positive).

2. Second, confidence interval can be entirely below zero. In this case we do not reject the null. However, in this case I think it is fine to say that we "accept the null", because we could consider $H_1$ as another null and reject that one.

3. Third, confidence interval can contain zero. Then we cannot reject $H_0$ and we cannot reject $H_1$ either, so there is nothing to accept.

So I would say that in one-sided situations one can accept the null, yes. But we cannot accept it simply because we failed to reject it; there are three possibilities, not two.

(Exactly the same applies to tests of equivalence aka "two one sided tests" (TOST), tests of non-inferiority, etc. One can reject the null, accept the null, or obtain an inconclusive result.)

In contrast, when $H_0$ is a point null such as $H_0:\mu=0$, we can never accept it, because $H_1:\mu\ne 0$ does not constitute a valid null hypothesis.

(Unless $\mu$ can have only discrete values, e.g. must be integer; then it seems that we could accept $H_0:\mu=0$ because $H_1:\mu\in\mathbb Z,\mu\ne 0$ now does constitute a valid null hypothesis. This is a bit of special case though.)

---

This issue was discussed some time ago in the comments under @gung's answer here: Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?

See also an interesting (and under-voted) thread Does failure to reject the null in Neyman-Pearson approach mean that one should "accept" it?, where @Scortchi explains that in the Neyman-Pearson framework some authors have no problem talking about "accepting the null". That is also what @Alexis means in the last paragraph of her answer here.

Answer 2

We never "accept the null hypothesis" (without also giving consideration to power and minimum relevant effect size). With a single hypothesis test, we pose a state of nature, $H_{0}$, and then answer some variation of the question "how unlikely are we to have observed the data underlying our test statistic, assuming $H_{0}$ (and our distributional assumption) is true?" We will then reject or fail to reject our $H_{0}$ based on a preferred Type I error rate, and draw a conclusion that is always about $H_{A}$… that is we found evidence to conclude $H_{A}$, or we did not find evidence to conclude $H_{A}$. We do not accept $H_{0}$ because we did not look for evidence for it. Absence of evidence (e.g., of a difference), is not the same thing as evidence of absence (e.g., of a difference). .

This is true for one-sided tests, just as it is for two-sided tests: we only look for evidence in favor of $H_{A}$ and find it, or do not find it.

If we only pose a single $H_{0}$ (without giving serious attention to both minimum relevant effect size, and statistical power), we are effectively making an a priori commitment to confirmation bias, because we have not looked for evidence for $H_{0}$, only evidence for $H_{A}$. Of course, we can (and, dare I say, should) pose null hypotheses for and against a position (relevance tests that combine tests for difference ($H_{0}^{+}$) with tests for equivalence ($H^{-}_{0}$) do just this).

It seems to me that there is no reason why you cannot combine inference from a one-sided test for inferiority with a one-sided test for non-inferiority to provide evidence (or lack of evidence) in both directions simultaneously.

Of course, if one is considering power and effect size, and one fails to reject $H_{0}$, but knows that there is (a) some minimum relevant effect size $\delta$, and (b) that their data are powerful enough to detect it for a given test, then one can interpret that as evidence of $H_{0}$.