Let the random variables $X \sim \mathcal{N} \left( \mu_1, \sigma^2_1 \right)$ and $Y \sim \mathcal{N} \left( \mu_2, \sigma^2_2 \right)$. Take the samples $x_1, \dots, x_{n_1} \in X$ and $y_1, \dots, y_{n_2} \in Y$ for some natural numbers $n_1$ and $n_2$. Define the sample mean $\bar{x} = \frac{1}{n_1} \sum_{i=1}^{n_1} x_i$, and the sample variance $s^2_1 = \frac{1}{n_1-1} \sum_{i=1}^{n_1} \left( x_i - \bar{x} \right)^2$, and define $\bar{y}$ and $s^2_2$ similarly.

Consider the $t$-test statistic for testing $H_0: \mu_1 - \mu_2 = \delta$: $$T = \frac{\left( \bar{x} - \bar{y} \right) - \delta}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}}.$$

Using the test statistic $T$, I wish to "prove" that $\mu_1 - \mu_2 = 0$. Of course, this can't be done directly since $T$ is continuous over $\delta$.

However, I've been given the information that there's some tolerance allowed so that I'm only trying to prove that $| \mu_1 - \mu_2 | < \epsilon$ for some given positive $\epsilon$. Therefore I wish to test the hypothesis $$\begin{cases} H_0 : | \mu_1 - \mu_2 | < \epsilon \\ H_1: | \mu_1 - \mu_2 | \geq \epsilon \end{cases}$$

At the end of the day, I want to be able to state that the null hypothesis $H_0$ is true with some confidence. To do this, if I fail to reject $H_0$, I will state that "I am $\beta$% confident that the mean difference is within the given tolerance of zero", where $\beta$ is the true negative rate.

There are problems here, though. I'm not sure how to perform the $t$-test when there's a composite null (unless I choose the $p$-value very conservatively by taking the largest $p$-value over the null), and, again, I'm not sure how to most reasonably evaluate the true negative rate over the composite alternative.

What should I do?

  • 5
    $\begingroup$ You might find useful material by searching for keywords TOST or "equivalence test." Although yours is a beautifully crafted and clear question, I suspect some version of it has been asked and answered (perhaps several times). $\endgroup$ – whuber Aug 12 '16 at 22:50
  • 3
    $\begingroup$ @Benjamin If you have access, try reading "A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability" by D. Schuirmann, Journal of Pharmacokinetics and Biopharmaceutics 15(6), 657-680, 1987. $\endgroup$ – mark999 Aug 12 '16 at 22:59
  • 1
    $\begingroup$ This sounds to me like an estimation problem disguised as a hypothesis-testing problem, because you want a confidence (or probability, or credibility, or something) at the end rather than a binary decision. I don't think there's any frequentist method that will help you here, but you could frame it in Bayesian terms and compute the posterior probability that $|μ_1 - μ_2| < ε$, or just compute the posterior distribution of $μ_1 - μ_2$. $\endgroup$ – Kodiologist Aug 12 '16 at 23:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.