I don't understand why hypothesis tests only test whether there is a difference in statistics, but not how big the difference is. Is there something wrong with the following approach?

Let $X_s = \{ X_1, ..., X_n \}$ and $ Y_s = \{ Y_1, ..., Y_n \}$ be two sets of samples. Let's say, the sample mean, $\mu_{X_s}$, of $X_s$ is larger than the sample mean, $\mu_{Y_s}$ of $Y_s$. I can test the statistical significance of the difference by testing the following hypotheses:

$H_0\colon \mu_X = \mu_Y \qquad \text{vs.} \qquad H_A\colon \mu_X > \mu_Y. $

I set the significance level at $0.05$, conduct an appropriate test, and get a very small $p$-value of, say, $0.00001$. I'm now pretty confident that the population means are different.

My results would be stronger if I could show by how much $\mu_X$ is bigger than $\mu_Y$. Why can't I successively test the following hypotheses

$H_0^0\colon \mu_X = \mu_Y + 0 \qquad \text{vs.} \qquad H_A^0\colon \mu_X > \mu_Y + 0 $ $H_0^1\colon \mu_X = \mu_Y + 1 \qquad \text{vs.} \qquad H_A^1\colon \mu_X > \mu_Y + 1 $ $H_0^2\colon \mu_X = \mu_Y + 2 \qquad \text{vs.} \qquad H_A^2\colon \mu_X > \mu_Y + 2 $


$H_0^k\colon \mu_X = \mu_Y + k \qquad \text{vs.} \qquad H_A^k\colon \mu_X > \mu_Y + k $

As long as my $p$-value is below $0.05$, I could reject the null hypothesis in favor of the alternative, and claim a stronger result, right? For example, if for $k = 3$ the $p$-value is $0.046$ and for $k = 4$ it is $0.061$ can I not claim that there is statistically significant evidence that $\mu_X > \mu_Y + 3$?

Now, I understand that the $p$-value is the probability of rejecting the null hypothesis when it is true. So if I conduct $k > 1$ independent tests, and always get $p$-values less than $0.05$, the probability of never rejecting a true null hypothesis is greater than $0.95^k$, which approaches zero as $k$ grows, so that's a bad idea. But the tests above are clearly not independent and therefore this argument doesn't hold, at least not in this form. What am I missing?


You're running head-first into a big wall of misunderstanding about hypothesis testing :-)

First, you want to do multiple hypothesis tests, so you need to correct your $p$-values/significance levels: start reading here.

Second, the hypothesis testing framework is as follows:

  1. Formulate your question.
  2. Translate your question into a null hypothesis/alternative problem.
  3. Make assumptions about the distribution of your prospective data and choose the corresponding tests.
  4. Decide on the significance level $\alpha$ of making an error of the first kind, the power of the test you want to achieve and choose your sample size.
  5. Conduct the experiment!
  6. Calculate your test statistic and $p$-value based on the data from 5).
  7. Reject your null hypothesis with error $\alpha$ if $p\leq \alpha$.

See that you have basically decided on everything before you do the experiment.

Now in your case, you would need to decide on $n$ and $\alpha$ beforehand, but also the maximum difference of interest $k$. Since the tests are dependent, you could use the simple Bonferroni correction $\alpha/(k+1)$, which means you have a more conservative significance level for every single one of your tests.

  • $\begingroup$ Thanks a lot for your answer. I’m aware of the problem of testing multiple hypotheses (tried hinting at it with the last paragraph), but can’t see how it applies to the described situation. I don’t want to test different symptoms but rather the magnitude of one symptom. It’s my understanding that hypothesis tests allows one to assess how likely the observed data is, given a particular assumption about the true population, i.e., the null hypothesis. Is this not correct? I’d much rather have a better understanding of the underlying assumptions than blindly follow a recipe. Thanks again! $\endgroup$ – red Apr 11 '19 at 9:39
  • $\begingroup$ That's kind of the problem with hypothesis testing -- it is a recipe... You're almost right about what hypothesis testing allows you to do: it allows you to assess how likely it is to observe THIS data or even more extreme data GIVEN that your null hypothesis is true. if this probability (the p-value) is very low, it's reasonable to reject the null hypothesis. $\endgroup$ – Edgar Apr 11 '19 at 9:46
  • $\begingroup$ Unfortunately, this doesn't allow you to do tests until you find the "correct difference" in means or something similar. You would need to decide beforehand what difference you want to test and include this difference in the null hypothesis. $\endgroup$ – Edgar Apr 11 '19 at 9:49
  • $\begingroup$ Ok. Do you know of any method that does allow to find the "correct difference" with a certain confidence? $\endgroup$ – red Apr 12 '19 at 7:59
  • $\begingroup$ No, sorry... Maybe Bayesian statistics could help? BTW, what's your application or are you just asking out of curiosity? $\endgroup$ – Edgar Apr 15 '19 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.