Hypothesis Testing when the aim is to claim that the two population mean (or any other statistic) are same

Question

In general, when doing hypothesis testing, we want to claim that the concerned statistic is different between the two population. Hence we assume as our null hypothesis that there is no difference in the two population and the alternative hypothesis that there is a difference between the two populations which we want to claim. According to me, this is similar to "proof by contradiction" in mathematics.

But what if my aim is to claim that the two populations are similar. Let us say that I want to show that my model is correct, and to show this I am claiming that the my model data and experimental data are similar. In this case, if I want to do hypothesis testing, I cannot (or should not) assume that my null hypothesis is that there is no difference in populations. Because it will be akin to "proof by assumption" which is not permitted in mathematics. And if I do not assume that my null hypothesis is "no difference in populations", then my null distribution won't come out to be the conventional distributions like normal, t distribution, chi-square etc. and I won't be able to perform the standardized tests.

And even if I assume my null hypothesis to be no difference in the populations. Then by choosing a lesser alpha value like 0.05, I am making very easy for my claim to pass the test. The probability of my claim will pass the test will be 95%. So should my methodology be to assume the "conventional" null hypothesis of no difference but then choose a high alpha value like 0.95 for testing my claim. Is this a correct way to think about this or are there other standard ways when the aim is to claim that there is no difference in the populations.

There should be some standard way to tackle this but I think I lack the terminology to search for the relevant information.

Answer 1

If your aim is to claim evidence that two (or more) populations are similar, for example, have similar means, then you are unequivocally looking for some variation of equivalence testing. In a two-sample test, the basic idea is expressed using this general 'negativist' null hypothesis:

$\text{H}_{0}^{-}\text{: }|\theta| \ge \Delta\text{, with}$
$\text{H}_{\text{A}}^{-}\text{: }|\theta| < \Delta\text{, where, for example,}$
$\theta = \mu_1 - \mu_2\text{, etc.}$

There are different ways of constructing test statistics for such a null hypothesis. A relatively straightforward one is called Two One-Sided Tests for equivalence (TOST). TOST works by recognizing that the general $\text{H}_{0}^{-}$, can be re-expressed as two one-sided null hypotheses thus:

$\text{H}_{01}^{-}\text{: }\theta \ge \Delta\text{ and}$
$\text{H}_{02}^{-}\text{: }\theta \le -\Delta$

with the corresponding alternate hypotheses:

$\text{H}_{\text{A}1}^{-}\text{: }\theta < \Delta\text{ and}$
$\text{H}_{\text{A}2}^{-}\text{: }\theta > -\Delta$

The logic is that if you reject $\text{H}_{01}^{-}$, then you conclude that you found evidence that $\theta$ is less than $\Delta$. Likewise, if you reject $\text{H}_{02}^{-}$, the you conclude that you found evidence that $\theta$ is greater than $-\Delta$. If you reject them both, then you found evidence that $\theta$ lies in the range $-\Delta, \Delta$. One could construct test statistics $t_1 = \frac{\Delta - (\bar{x}_1 - \bar{x}_2)}{s_{\bar{x}}}$ (with $p_1 = P(T_{\nu} > t_1)$, and $t_2 = \frac{(\bar{x}_1 - \bar{x}_2) + \Delta}{s_{\bar{x}}}$ (with $p_2 = P(T_{\nu} > t_2)$) to test $\text{H}_{01}^{-}$ and $\text{H}_{02}^{-}$ (I construct both statistics with upper tail rejection regions here).

If $\Delta$ is a researcher expression of something like 'the smallest difference between $\mu_1$ and $\mu_2$ that I care about', then the usefulness of equivalence tests becomes evident.

There are more sophisticated single-test statistic ways to test $\text{H}_{0}^{-}$. For a good text, see Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (Second Edition). Chapman and Hall/CRC Press.

There is another approach, where one compares the $\alpha \%$ (e.g., $90\%$) confidence interval for $\mu_1 - \mu_2$, and if it does not extend outside the range $-\Delta, \Delta$, one can conclude one has evidence for equivalence at the $\frac{\alpha}{2}$ (e.g., $95\%$) level. (The 'gain' in confidence is due to $\text{H}_{01}^{-}$ and $\text{H}_{02}^{-}$ being mutually exclusive… they cannot both be true).