2
$\begingroup$

In general, when doing hypothesis testing, we want to claim that the concerned statistic is different between the two population. Hence we assume as our null hypothesis that there is no difference in the two population and the alternative hypothesis that there is a difference between the two populations which we want to claim. According to me, this is similar to "proof by contradiction" in mathematics.

But what if my aim is to claim that the two populations are similar. Let us say that I want to show that my model is correct, and to show this I am claiming that the my model data and experimental data are similar. In this case, if I want to do hypothesis testing, I cannot (or should not) assume that my null hypothesis is that there is no difference in populations. Because it will be akin to "proof by assumption" which is not permitted in mathematics. And if I do not assume that my null hypothesis is "no difference in populations", then my null distribution won't come out to be the conventional distributions like normal, t distribution, chi-square etc. and I won't be able to perform the standardized tests.

And even if I assume my null hypothesis to be no difference in the populations. Then by choosing a lesser alpha value like 0.05, I am making very easy for my claim to pass the test. The probability of my claim will pass the test will be 95%. So should my methodology be to assume the "conventional" null hypothesis of no difference but then choose a high alpha value like 0.95 for testing my claim. Is this a correct way to think about this or are there other standard ways when the aim is to claim that there is no difference in the populations.

There should be some standard way to tackle this but I think I lack the terminology to search for the relevant information.

$\endgroup$
4
  • 5
    $\begingroup$ From a classical hypothesis testing perspective, it's not possible to test that two populations are the same, because they could be infinitesimally different. The solution is to evaluate whether the two populations are within some epsilon of each other. A good term to google with is "statistical equivalence testing", which is prominent in the field of measurement systems analysis. $\endgroup$ Jun 13, 2022 at 16:01
  • 1
    $\begingroup$ you cannot prove that your model is "correct", because no model is fully correct. you can only show that your model is a valid approximation of reality. besides, there is not "proving" in statistics. $\endgroup$
    – carlo
    Jun 13, 2022 at 16:01
  • 2
    $\begingroup$ stats.stackexchange.com/search?q=TOST $\endgroup$
    – whuber
    Jun 13, 2022 at 16:03
  • $\begingroup$ Just to emphasize that the epsilon in "whether the two populations are within some epsilon of each other" (in @JohnMadden 's comment) is essential. You, your subject matter, and the audience you're trying to convince need to set that value. And how close is close might also vary over time (due to new knowledge, politics, etc.). If there is no consensus as to the size and characterization of epsilon - as it might be multidimensional - then you still need to set a value and say "If epsilon is x, then this is what the data tells us." $\endgroup$
    – JimB
    Jun 23, 2022 at 22:53

1 Answer 1

3
$\begingroup$

If your aim is to claim evidence that two (or more) populations are similar, for example, have similar means, then you are unequivocally looking for some variation of equivalence testing. In a two-sample test, the basic idea is expressed using this general 'negativist' null hypothesis:

$\text{H}_{0}^{-}\text{: }|\theta| \ge \Delta\text{, with}$
$\text{H}_{\text{A}}^{-}\text{: }|\theta| < \Delta\text{, where, for example,}$
$\theta = \mu_1 - \mu_2\text{, etc.}$

There are different ways of constructing test statistics for such a null hypothesis. A relatively straightforward one is called Two One-Sided Tests for equivalence (TOST). TOST works by recognizing that the general $\text{H}_{0}^{-}$, can be re-expressed as two one-sided null hypotheses thus:

$\text{H}_{01}^{-}\text{: }\theta \ge \Delta\text{ and}$
$\text{H}_{02}^{-}\text{: }\theta \le -\Delta$

with the corresponding alternate hypotheses:

$\text{H}_{\text{A}1}^{-}\text{: }\theta < \Delta\text{ and}$
$\text{H}_{\text{A}2}^{-}\text{: }\theta > -\Delta$

The logic is that if you reject $\text{H}_{01}^{-}$, then you conclude that you found evidence that $\theta$ is less than $\Delta$. Likewise, if you reject $\text{H}_{02}^{-}$, the you conclude that you found evidence that $\theta$ is greater than $-\Delta$. If you reject them both, then you found evidence that $\theta$ lies in the range $-\Delta, \Delta$. One could construct test statistics $t_1 = \frac{\Delta - (\bar{x}_1 - \bar{x}_2)}{s_{\bar{x}}}$ (with $p_1 = P(T_{\nu} > t_1)$, and $t_2 = \frac{(\bar{x}_1 - \bar{x}_2) + \Delta}{s_{\bar{x}}}$ (with $p_2 = P(T_{\nu} > t_2)$) to test $\text{H}_{01}^{-}$ and $\text{H}_{02}^{-}$ (I construct both statistics with upper tail rejection regions here).

If $\Delta$ is a researcher expression of something like 'the smallest difference between $\mu_1$ and $\mu_2$ that I care about', then the usefulness of equivalence tests becomes evident.

There are more sophisticated single-test statistic ways to test $\text{H}_{0}^{-}$. For a good text, see Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (Second Edition). Chapman and Hall/CRC Press.

There is another approach, where one compares the $\alpha \%$ (e.g., $90\%$) confidence interval for $\mu_1 - \mu_2$, and if it does not extend outside the range $-\Delta, \Delta$, one can conclude one has evidence for equivalence at the $\frac{\alpha}{2}$ (e.g., $95\%$) level. (The 'gain' in confidence is due to $\text{H}_{01}^{-}$ and $\text{H}_{02}^{-}$ being mutually exclusive… they cannot both be true).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.