Using successive hypothesis tests to test the magnitude of the difference

Question

I don't understand why hypothesis tests only test whether there is a difference in statistics, but not how big the difference is. Is there something wrong with the following approach?

Let $X_s = \{ X_1, ..., X_n \}$ and $ Y_s = \{ Y_1, ..., Y_n \}$ be two sets of samples. Let's say, the sample mean, $\mu_{X_s}$, of $X_s$ is larger than the sample mean, $\mu_{Y_s}$ of $Y_s$. I can test the statistical significance of the difference by testing the following hypotheses:

$H_0\colon \mu_X = \mu_Y \qquad \text{vs.} \qquad H_A\colon \mu_X > \mu_Y. $

I set the significance level at $0.05$, conduct an appropriate test, and get a very small $p$-value of, say, $0.00001$. I'm now pretty confident that the population means are different.

My results would be stronger if I could show by how much $\mu_X$ is bigger than $\mu_Y$. Why can't I successively test the following hypotheses

$H_0^0\colon \mu_X = \mu_Y + 0 \qquad \text{vs.} \qquad H_A^0\colon \mu_X > \mu_Y + 0 $ $H_0^1\colon \mu_X = \mu_Y + 1 \qquad \text{vs.} \qquad H_A^1\colon \mu_X > \mu_Y + 1 $ $H_0^2\colon \mu_X = \mu_Y + 2 \qquad \text{vs.} \qquad H_A^2\colon \mu_X > \mu_Y + 2 $

$\vdots$

$H_0^k\colon \mu_X = \mu_Y + k \qquad \text{vs.} \qquad H_A^k\colon \mu_X > \mu_Y + k $

As long as my $p$-value is below $0.05$, I could reject the null hypothesis in favor of the alternative, and claim a stronger result, right? For example, if for $k = 3$ the $p$-value is $0.046$ and for $k = 4$ it is $0.061$ can I not claim that there is statistically significant evidence that $\mu_X > \mu_Y + 3$?

Now, I understand that the $p$-value is the probability of rejecting the null hypothesis when it is true. So if I conduct $k > 1$ independent tests, and always get $p$-values less than $0.05$, the probability of never rejecting a true null hypothesis is greater than $0.95^k$, which approaches zero as $k$ grows, so that's a bad idea. But the tests above are clearly not independent and therefore this argument doesn't hold, at least not in this form. What am I missing?

Answer 1

You're running head-first into a big wall of misunderstanding about hypothesis testing :-)

First, you want to do multiple hypothesis tests, so you need to correct your $p$-values/significance levels: start reading here.

Second, the hypothesis testing framework is as follows:

1. Formulate your question.
2. Translate your question into a null hypothesis/alternative problem.
3. Make assumptions about the distribution of your prospective data and choose the corresponding tests.
4. Decide on the significance level $\alpha$ of making an error of the first kind, the power of the test you want to achieve and choose your sample size.
5. Conduct the experiment!
6. Calculate your test statistic and $p$-value based on the data from 5).
7. Reject your null hypothesis with error $\alpha$ if $p\leq \alpha$.

See that you have basically decided on everything before you do the experiment.

Now in your case, you would need to decide on $n$ and $\alpha$ beforehand, but also the maximum difference of interest $k$. Since the tests are dependent, you could use the simple Bonferroni correction $\alpha/(k+1)$, which means you have a more conservative significance level for every single one of your tests.