I don't understand why hypothesis tests only test whether there is a difference in statistics, but not how big the difference is. Is there something wrong with the following approach?
Let $X_s = \{ X_1, ..., X_n \}$ and $ Y_s = \{ Y_1, ..., Y_n \}$ be two sets of samples. Let's say, the sample mean, $\mu_{X_s}$, of $X_s$ is larger than the sample mean, $\mu_{Y_s}$ of $Y_s$. I can test the statistical significance of the difference by testing the following hypotheses:
$H_0\colon \mu_X = \mu_Y \qquad \text{vs.} \qquad H_A\colon \mu_X > \mu_Y. $
I set the significance level at $0.05$, conduct an appropriate test, and get a very small $p$-value of, say, $0.00001$. I'm now pretty confident that the population means are different.
My results would be stronger if I could show by how much $\mu_X$ is bigger than $\mu_Y$. Why can't I successively test the following hypotheses
$H_0^0\colon \mu_X = \mu_Y + 0 \qquad \text{vs.} \qquad H_A^0\colon \mu_X > \mu_Y + 0 $ $H_0^1\colon \mu_X = \mu_Y + 1 \qquad \text{vs.} \qquad H_A^1\colon \mu_X > \mu_Y + 1 $ $H_0^2\colon \mu_X = \mu_Y + 2 \qquad \text{vs.} \qquad H_A^2\colon \mu_X > \mu_Y + 2 $
$\vdots$
$H_0^k\colon \mu_X = \mu_Y + k \qquad \text{vs.} \qquad H_A^k\colon \mu_X > \mu_Y + k $
As long as my $p$-value is below $0.05$, I could reject the null hypothesis in favor of the alternative, and claim a stronger result, right? For example, if for $k = 3$ the $p$-value is $0.046$ and for $k = 4$ it is $0.061$ can I not claim that there is statistically significant evidence that $\mu_X > \mu_Y + 3$?
Now, I understand that the $p$-value is the probability of rejecting the null hypothesis when it is true. So if I conduct $k > 1$ independent tests, and always get $p$-values less than $0.05$, the probability of never rejecting a true null hypothesis is greater than $0.95^k$, which approaches zero as $k$ grows, so that's a bad idea. But the tests above are clearly not independent and therefore this argument doesn't hold, at least not in this form. What am I missing?