6
$\begingroup$

Given two (numeric) samples I would like to show that there is not a significant difference between the two means $\mu_{1}$ and $\mu_{2}$.

If my goal was to show a significant difference I would formulate the $t$-test as follows:

(1) $H_{0}: \mu_{1} = \mu_{2}$ vs $H_{1}: \mu_{1} \neq \mu_{2}$

I learned in school that the null hypothesis should always represent the "common" belief and the alternative hypothesis should represent the change that I would like to show.

So then if my goal is to show that there is not a significant difference between both means, should I formulate the test like this?

(2) $H_{0}: \mu_{1} \neq \mu_{2}$ vs $H_{1}: \mu_{1} = \mu_{2}$

Or can I use the first test (1) and when I am not able to reject the null hypothesis say that there is not a significant difference?

$\endgroup$
  • 1
    $\begingroup$ The alternative hypothesis indicates what an extreme result might look like. The problem with your (2) formulation is that this would be a difference in means close to $0$; so if you took a commonly used significance level of $5\%$ then the power of the test (its ability to reject the null hypothesis when it is false) would never be above $5\%$ no matter how large the sample size. This is not good $\endgroup$ – Henry Jan 9 at 15:07
  • $\begingroup$ "I learned in school that the null hypothesis should always represent the "common" belief and the alternative hypothesis should represent the change that I would like to show." Then you were taught to commit to confirmation bias as a mode of scientific inquiry. $\endgroup$ – Alexis Jan 10 at 19:09
11
$\begingroup$

You cannot use the first test in the way you describe, because failure to reject in the first test only says that you were unable to reject $H_0$ nothing more than that. It is like only being given the information that "the prosecutor was unable to provide the jury with enough evidence to secure a conviction" - that does not tell you that the suspect is innocent.

The second test is not usable in practice, because no matter how much data you have, you cannot exclude the possibility of very small differences.

What you can do is to look at $$H_{0}: |\mu_{1} - \mu_{2}|>\delta \text{ vs }H_{1}: |\mu_{1} - \mu_{2}| \leq \delta,$$ i.e. try to reject the null hypothesis that the absolute size of the difference is greater than some difference $\delta>0$. $\delta$ would be chosen e.g. so that any difference smaller than that is for all (or your specific) practical purposes irrelevant.

$\endgroup$
-1
$\begingroup$

I learned in school that the null hypothesis should always represent the "common" belief and the alternative hypothesis should represent the change that I would like to show.

That is not accurate explanation of the null hypothesis. The null hypothesis is simply a hypothesis that consists of a specific distribution from which probabilities can be calculated. The reason we use $\mu_1=\mu_2$ as the null hypothesis has nothing to do with whether this is the "common" belief. It's used as the null hypothesis because if we hypothesize that the mean is a specific value, then given a particular set of data we can calculate the probability of seeing that data. We can't use $\mu \neq \mu_2$ as our null hypothesis because there's no way to calculate p-values based simply on the hypothesis that the means aren't equal to a particular value. Consider the following problem:

The weights of apples have a standard deviation of 5 grams. The mean is not equal to 100. What is the probability of seeing an apple with a weight of 110 grams?

There's no way to answer that, because simply being told what the mean isn't is not enough to calculate probabilities.

Björn suggests testing the hypothesis that the difference in means is greater than some $\delta_0$. How that would work is to take the null hypothesis as that the difference is equal to exactly $\delta_0$. Then once you have the data, you can calculate the p-value given that $\delta_0$. Call that $p_{\delta_0}$. If the difference in sample means is less than $\delta_0$, then the the p-value would have been even smaller than $p_{\delta_0}$ if we had chosen $\delta$ to be larger than $\delta_0$. We reject the null if the p-value is less than $\alpha$, so if we're rejecting under that null, that means that $p_{\delta_0} < \alpha$. And since $p_{\delta}<p_{\delta_0}$ for any $\delta>\delta_0$, we can conclude that $p_{\delta}<\alpha$ for any $\delta>\delta_0$. Thus, we can not only reject this null of $\delta_0$, but we can reject any null with a larger $\delta$. It is only because of this ability to get an upper bound on p that we don't need a specific value for $\delta$. If we just take "$\delta$ is larger than zero" as our null hypothesis, without any lower bound for $\delta$, then there is no upper bound for p, and so we cannot conclude that it is lower than $\alpha$.

$\endgroup$
  • 1
    $\begingroup$ You seem quite confused about how two one-sided tests for equivalence work: "How that would work is to take the null hypothesis as that the difference is equal to exactly $\delta_{0}$." is not remotely close to these procedures. $\endgroup$ – Alexis Jan 10 at 19:12
  • $\begingroup$ @Alexis That is the rigorous mathematical theoretical foundation of the process. Certainly, there are people doing statistics in the field that are not engaging in full rigor. $\endgroup$ – Acccumulation Jan 10 at 19:15

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.