What will be the null hypothesis in this study?

Question

The disease X can be treated with either drug A or drug B. Doctors have used one of these two drugs for several years. From their experience, they believe that mortality with both drugs is similar.

A researcher desires to formally compare the two drugs. The researcher plans to retrospectively analyze the mortality in the patients treated with the two drugs. The researcher's study hypothesis states that the mortality with drug A and drug B is not different. The researcher obtains a P-value >0.05 for this comparison and concludes that both the drugs are comparable.

Now, I know that in a clinical trial, such a study hypothesis (stating no difference between two arms) is not acceptable. The establishment of comparability of the two drugs would need an equivalence trial. More commonly, researchers would use a superiority trial. The study hypothesis of the superiority trial would state that one of the drugs is better than the other. The null hypothesis would state that the drugs are not different.

However, in the setting of a retrospective study (when experience makes us believe that the drugs may be comparable), (1) Can we use a study hypothesis that states that the drugs are similar? (2) If (1) is acceptable, can the researcher conclude that the drugs are comparable when the P-value is not statistically significant and fails to reject the null hypothesis? (3) Should we use a study hypothesis which states that one drug is better than the other (similar to a clinical trial) disregarding our experience?

Answer 1

1. Yes: The general form of the "negativist" null hypothesis is $\text{H}_{0}^{-}\text{: }|\mu_{\text{A}} - \mu_{\text{B}}| \ge \Delta$, with $\text{H}_{\text{A}}^{-}\text{: }|\mu_{\text{A}} - \mu_{\text{B}}| < \Delta$. In plain language, the magnitude of the difference in means of $\text{A}$ and $\text{B}$ is greater than $\Delta$, where $\Delta$ is the smallest difference between the two means that you care about. In TOST, the general form does not have a test statistic, so it has to be reexpressed with two specific one-sided null hypotheses: $\text{H}_{01}^{-}\text{: }\mu_{\text{A}} - \mu_{\text{B}} \ge \Delta$ or $\text{H}_{02}^{-}\text{: }\mu_{\text{A}} - \mu_{\text{B}} \le -\Delta$. If you reject $\text{H}_{01}^{-}$, then you conclude that you found evidence that $\mu_{\text{A}} - \mu_{\text{B}} < \Delta$. And if you reject $\text{H}_{02}^{-}$, then you conclude that you found evidence that $\mu_{\text{A}} - \mu_{\text{B}} > -\Delta$. However, if you reject both $\text{H}_{01}^{-}$ and $\text{H}_{02}^{-}$, then you have rejected $\text{H}_{0}^{-}$, and found evidence of the general negativist alternative hypothesis… that is that the magnitude of the differences in means between $\text{A}$ and $\text{B}$ is too small to care about.

2. No: You perform two one-sided tests, with a separate test statistic for each one-sided test. For example, $t_{1} = \frac{(\mu_{\text{A}} - \mu_{\text{B}}) - \Delta}{s_{\mu_{\text{A}} - \mu_{\text{B}}}}$, with $p_1 = P(T_{\nu} \le t_{1})$, and $t_{2} = \frac{(\mu_{\text{A}} - \mu_{\text{B}}) + \Delta}{s_{\mu_{\text{A}} - \mu_{\text{B}}}}$, with $p_2 = P(T_{\nu} \ge t_{2})$. Both these null hypotheses are rejected at the $\alpha$ level. There is a more mathematically complex way to create a single test statistic to directly test $\text{H}_{0}^{-}$ and $p$ value for that test statistic called "uniformly most powerful tests" (for this kind of equivalence test, see Wellek, Stefan. Testing Statistical Hypotheses of Equivalence and Noninferiority. Second Edition. Chapman and Hall/CRC Press, 2010).

3. No. However, there are separate tests for non-inferiority and non-superiority ($\text{H}_{01}^{-}$ is the null which rejecting provides evidence of non-superiority; $\text{H}_{02}^{-}$ is the null which rejecting provides evidence of non-inferiority).

Nuances:

• My answer above assumes a symmetrical equivalence region—the magnitude of the difference between groups is either larger than $\Delta$, or less than $-\Delta$. However, this need not be the case, and there might be $\Delta_{\text{upper}} \ne |\Delta_{\text{lower}}|$, where $\Delta_{\text{lower}}<0$. If this is so, then the test statistics become $t_{1} = \frac{(\mu_{\text{A}} - \mu_{\text{B}}) - \Delta_{\text{upper}}}{s_{\mu_{\text{A}} - \mu_{\text{B}}}}$, and $t_{2} = \frac{(\mu_{\text{A}} - \mu_{\text{B}}) - \Delta_{\text{lower}}}{s_{\mu_{\text{A}} - \mu_{\text{B}}}}$.

• My answer above assumes you are performing a test of absolute difference. This is not always the case in clinical trials, where relative differences may also be used (e.g., ORs, RRs, etc.). When this is the case, the inferential target has an asymmetric scale, so a "symmetric" equivalence threshold then becomes $\Delta_{\text{lower}} = \frac{1}{\Delta_{\text{upper}}}$ (and to be super explicit: $\Delta_{\text{lower}} >0$). Of course one could also use $\Delta_{\text{lower}} \ne \frac{1}{\Delta_{\text{upper}}}$ if one is choosing an "asymmetric" equivalence region analogous to my first nuanced point.

Answer 2

1. Yes. This is called equivalence testing. The easiest equivalence test to understand is called two one-sided tests (TOST). Briefly, TOST uses one-sided tests to show that neither drug is much better than the other, bounding the difference in some region the investigator deems practically equivalent.

2. No. This is a common misinterpretation of what a p-value means. A high p-value can result from a test having inadequate power to detect a difference of interest. In other words, if you look for something and don’t find it, it doesn’t count unless you’ve looked hard enough (where “hard enough” is quantified by a power calculation).

3. No. There are methods, like TOST, that allow you to test exactly what you want to test.