9
$\begingroup$

In a usual t-test of means, using the usual hypothesis testing methods, we either reject the null or fail to reject the null but we never accept the null. One reason for this is that if we got more evidence, the same effect size would become significant.

But what happens in a noninferiority test?

That is:

$$H_0: \mu_1 - \mu_0 \le x$$

vs.

$$H_1: \mu_1 - \mu_0 > x$$

where $x$ is some amount that we regard as essentially the same. So, if we reject the null we say that $\mu_1$ is greater than $\mu_0$ by at least $x$. We fail to reject the null if there is insufficient evidence.

If the effect size is $x$ or greater, then this is analogous to the regular t-test. But what if the effect size is less than $x$ in the sample we have? Then, if we increased the sample size and kept the same effect, it would stay nonsignificant. Can we, therefore, accept the null in this case?

$\endgroup$
  • 1
    $\begingroup$ Are your hypotheses mixed up? Normally, for a NI test the null hypothesis is that the difference is greater than x, while the alternative is that it is les or equal than x. I guess it depends on the order of your difference scale. $\endgroup$ – Björn Mar 7 '17 at 5:31
  • $\begingroup$ Hi @Björn it would depend on whether higher is worse or higher is better. $\endgroup$ – Peter Flom Mar 7 '17 at 12:13
  • 1
    $\begingroup$ Is it the same as asking if one can accept the null in one-sided tests? There was some discussion of it in the comments to stats.stackexchange.com/a/85914. $\endgroup$ – amoeba Mar 8 '17 at 23:48
  • 2
    $\begingroup$ @amoeba I think Peter presents a fascinating argument (+1), perhaps more akin to a paradox. One conventional explanations of why we don't "accept H0" one sometimes hears is "if we got more evidence, the same effect size would become significant". But following that logic as Peter does, we either come to the conclusion that in some situations we should "accept H0", or if we do not, that the "reason" is actually wrong, and not why we do it at all. I believe you are correct - his argument would apply to one-sided t-tests too, since a negative effect size remains insignificant as n increases $\endgroup$ – Silverfish Mar 9 '17 at 0:25
  • 1
    $\begingroup$ Yes, I agree: the linked answer does not answer your question. I only provided the link because there was a related discussion in the comments there. $\endgroup$ – amoeba Mar 9 '17 at 11:55
6
+200
$\begingroup$

Your logic applies in exactly the same way to the good old one-sided tests (i.e. with $x=0$) that may be more familiar to the readers. For concreteness, imagine we are testing the null $H_0:\mu\le0$ against the alternative that $\mu$ is positive. Then if true $\mu$ is negative, increasing sample size will not yield a significant result, i.e., to use your words, it is not true that "if we got more evidence, the same effect size would become significant".

If we test $H_0:\mu\le 0$, we can have three possible outcomes:

  1. First, $(1-\alpha)\cdot100\%$ confidence interval can be entirely above zero; then we reject the null and accept the alternative (that $\mu$ is positive).

  2. Second, confidence interval can be entirely below zero. In this case we do not reject the null. However, in this case I think it is fine to say that we "accept the null", because we could consider $H_1$ as another null and reject that one.

  3. Third, confidence interval can contain zero. Then we cannot reject $H_0$ and we cannot reject $H_1$ either, so there is nothing to accept.

So I would say that in one-sided situations one can accept the null, yes. But we cannot accept it simply because we failed to reject it; there are three possibilities, not two.

(Exactly the same applies to tests of equivalence aka "two one sided tests" (TOST), tests of non-inferiority, etc. One can reject the null, accept the null, or obtain an inconclusive result.)

In contrast, when $H_0$ is a point null such as $H_0:\mu=0$, we can never accept it, because $H_1:\mu\ne 0$ does not constitute a valid null hypothesis.

(Unless $\mu$ can have only discrete values, e.g. must be integer; then it seems that we could accept $H_0:\mu=0$ because $H_1:\mu\in\mathbb Z,\mu\ne 0$ now does constitute a valid null hypothesis. This is a bit of special case though.)


This issue was discussed some time ago in the comments under @gung's answer here: Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?

See also an interesting (and under-voted) thread Does failure to reject the null in Neyman-Pearson approach mean that one should "accept" it?, where @Scortchi explains that in the Neyman-Pearson framework some authors have no problem talking about "accepting the null". That is also what @Alexis means in the last paragraph of her answer here.

$\endgroup$
  • $\begingroup$ If the $(1-\alpha)$ confidence interval is entirely above zero then reject the null that $\mu\leq 0$: that's a test with a worst-case size of $\frac{\alpha}{2}$. If the $(1-\alpha)$ confidence interval is entirely below zero then reject the null that $\mu>0$: that's a test with a worst-case size of $\frac{\alpha}{2}$. By combining the two tests you can maintain a worst-case size of $\frac{\alpha}{2}$ because the two nulls are mutually exclusive. So the three outcomes can be described in terms of accepting one alternative, or another alternative, or rejecting neither null. $\endgroup$ – Scortchi Mar 16 '17 at 18:53
  • $\begingroup$ A two-tailed test can be thought of similarly as composed of two one-sided tests; but the alternatives aren't mutually exclusive, & the worst-case size is $\alpha$ (when $\mu=0$). $\endgroup$ – Scortchi Mar 16 '17 at 19:26
  • $\begingroup$ Thanks @Scortchi. Somehow I am not quite sure if you are agreeing or disagreeing with my answer. $\endgroup$ – amoeba Mar 16 '17 at 22:35
  • $\begingroup$ As $\mu\leq 0$ isn't accepted qua null in one test, but qua alternative in another, I feel "accepting the null" is needlessly confusing here; nevertheless your procedure should satisfy those with the urge to. What perhaps deserves greater emphasis in your answer is the difference between combining tests for non-inferiority vs inferiority & vice versa, and tests for superiority vs non-superiority (or the nil null) & inferiority vs non-inferiority (or the nil null). $\endgroup$ – Scortchi Mar 17 '17 at 10:22
  • $\begingroup$ @Scortchi The syntax of your last sentence is quite complicated: what exactly can (or can't) be combined and what exactly is the difference? I am not sure I understood you correctly, sorry. $\endgroup$ – amoeba Mar 17 '17 at 12:36
6
$\begingroup$

We never "accept the null hypothesis" (without also giving consideration to power and minimum relevant effect size). With a single hypothesis test, we pose a state of nature, $H_{0}$, and then answer some variation of the question "how unlikely are we to have observed the data underlying our test statistic, assuming $H_{0}$ (and our distributional assumption) is true?" We will then reject or fail to reject our $H_{0}$ based on a preferred Type I error rate, and draw a conclusion that is always about $H_{A}$… that is we found evidence to conclude $H_{A}$, or we did not find evidence to conclude $H_{A}$. We do not accept $H_{0}$ because we did not look for evidence for it. Absence of evidence (e.g., of a difference), is not the same thing as evidence of absence (e.g., of a difference). .

This is true for one-sided tests, just as it is for two-sided tests: we only look for evidence in favor of $H_{A}$ and find it, or do not find it.

If we only pose a single $H_{0}$ (without giving serious attention to both minimum relevant effect size, and statistical power), we are effectively making an a priori commitment to confirmation bias, because we have not looked for evidence for $H_{0}$, only evidence for $H_{A}$. Of course, we can (and, dare I say, should) pose null hypotheses for and against a position (relevance tests that combine tests for difference ($H_{0}^{+}$) with tests for equivalence ($H^{-}_{0}$) do just this).

It seems to me that there is no reason why you cannot combine inference from a one-sided test for inferiority with a one-sided test for non-inferiority to provide evidence (or lack of evidence) in both directions simultaneously.

Of course, if one is considering power and effect size, and one fails to reject $H_{0}$, but knows that there is (a) some minimum relevant effect size $\delta$, and (b) that their data are powerful enough to detect it for a given test, then one can interpret that as evidence of $H_{0}$.

$\endgroup$
  • 1
    $\begingroup$ Peter's question contained a particularly interesting point that this answer seems to skirt around: that one of the conventional explanations given of the standard "fail to reject H0" terminology is that e.g. in a t-test, if we got more evidence, the same effect size would become significant. But if this were the "real" reason we "fail to reject", his argument that we could "accept H0" in the circumstances he outlines seems (to me at least) to be a strong one - though I'm not sure I've seen it done other than casually, as a sort of statistical slang, rather than consciously and deliberately. $\endgroup$ – Silverfish Mar 9 '17 at 0:19
  • 1
    $\begingroup$ This answer restates the conventional position on "accepting H0" in a nice, clear, succinct way but doesn't seem to directly address the argument (or perhaps, paradox) at the heart of Peter's question. What do you think of the "we can't accept H0 because if we got more evidence, the same effect size would become significant" argument for the conventional terminology - is there some flaw in Peter's presentation or extension of it, or was the logic of the original argument invalid in the first place? $\endgroup$ – Silverfish Mar 9 '17 at 0:19
  • 1
    $\begingroup$ @Silverfish follow the link in my answer to "relevance tests" for more amplification of my critical resolution to the issue of "we can't accept H0 because if we got more evidence, the same effect size would become significant" $\endgroup$ – Alexis Mar 9 '17 at 1:27
  • 1
    $\begingroup$ @Alexis I have to agree with Silverfish. I appreciate your answer, but it doesn't address my central point, for the reason Silverfish enunciated. If we had N = 1,000,000 then pretty much any difference would be significant in standard setting. But in the noninferiority case, that isn't so. And even in TOST two sided, it isn't so. If the difference is less than the amount we deem important, then no N will make it sig. $\endgroup$ – Peter Flom Mar 9 '17 at 1:57
  • 1
    $\begingroup$ Apologies - my 1st comment was intended merely as a prelude to the 2nd (or more accurately, the 2nd was the overflow of the 1st!) and wasn't intended to raise a freestanding point of its own. The link was helpful, thanks. Your central point (which you put very nicely, both in your answer and your restatement) explains clearly why you disagree with Peter's conclusion. But I was curious where you felt the flaw was in his logic - or perhaps its premise. This is the bit that felt to me to not have been tackled directly. $\endgroup$ – Silverfish Mar 9 '17 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.