Using successive hypothesis tests to test the magnitude of the difference

Question

I don't understand why hypothesis tests only test whether there is a difference in statistics, but not how big the difference is. Is there something wrong with the following approach?

Let $X_s = \{ X_1, ..., X_n \}$ and $ Y_s = \{ Y_1, ..., Y_n \}$ be two sets of samples. Let's say, the sample mean, $\mu_{X_s}$, of $X_s$ is larger than the sample mean, $\mu_{Y_s}$ of $Y_s$. I can test the statistical significance of the difference by testing the following hypotheses:

$H_0\colon \mu_X = \mu_Y \qquad \text{vs.} \qquad H_A\colon \mu_X > \mu_Y. $

I set the significance level at $0.05$, conduct an appropriate test, and get a very small $p$-value of, say, $0.00001$. I'm now pretty confident that the population means are different.

My results would be stronger if I could show by how much $\mu_X$ is bigger than $\mu_Y$. Why can't I successively test the following hypotheses

$H_0^0\colon \mu_X = \mu_Y + 0 \qquad \text{vs.} \qquad H_A^0\colon \mu_X > \mu_Y + 0 $ $H_0^1\colon \mu_X = \mu_Y + 1 \qquad \text{vs.} \qquad H_A^1\colon \mu_X > \mu_Y + 1 $ $H_0^2\colon \mu_X = \mu_Y + 2 \qquad \text{vs.} \qquad H_A^2\colon \mu_X > \mu_Y + 2 $

$\vdots$

$H_0^k\colon \mu_X = \mu_Y + k \qquad \text{vs.} \qquad H_A^k\colon \mu_X > \mu_Y + k $

As long as my $p$-value is below $0.05$, I could reject the null hypothesis in favor of the alternative, and claim a stronger result, right? For example, if for $k = 3$ the $p$-value is $0.046$ and for $k = 4$ it is $0.061$ can I not claim that there is statistically significant evidence that $\mu_X > \mu_Y + 3$?

Now, I understand that the $p$-value is the probability of rejecting the null hypothesis when it is true. So if I conduct $k > 1$ independent tests, and always get $p$-values less than $0.05$, the probability of never rejecting a true null hypothesis is greater than $0.95^k$, which approaches zero as $k$ grows, so that's a bad idea. But the tests above are clearly not independent and therefore this argument doesn't hold, at least not in this form. What am I missing?

Answer 1

It seems to me that you do not want to use the hypothesis test framework at all. Instead look at the data as evidence and determine the relationship between hypothesised difference sizes and the evidential support.

The procedure you describe is analogous to a little-used 'consonance curve' which is a plot of the hypothesised differences and their p-values. The function is not often (ever?) used, perhaps because the p-values do not have a simple interpretation, but it is worth looking at. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-01105-9

What you should want is a likelihood function that depicts in a straightforwardly interpreted manner the evidential support of the data for the various possible differences.

Answer 2

Your suggestion of testing $k$ different hypotheses and finding those that are not significant (not rejected) and those that are significant (rejected) is the definition of a confidence interval. See inverting a hypothesis test. See my post here where I invert a likelihood ratio test. This is precisely what is done to construct a Wald confidence interval but it is so deceptively simple that this detail is often overlooked. A confidence interval is a set of plausible true values of $\mu_X-\mu_Y$ given the observed data $\boldsymbol{X}=\boldsymbol{x}$, $\boldsymbol{Y}=\boldsymbol{y}$.

Answer 3

You're running head-first into a big wall of misunderstanding about hypothesis testing :-)

First, you want to do multiple hypothesis tests, so you need to correct your $p$-values/significance levels: start reading here.

Second, the hypothesis testing framework is as follows:

1. Formulate your question.
2. Translate your question into a null hypothesis/alternative problem.
3. Make assumptions about the distribution of your prospective data and choose the corresponding tests.
4. Decide on the significance level $\alpha$ of making an error of the first kind, the power of the test you want to achieve and choose your sample size.
5. Conduct the experiment!
6. Calculate your test statistic and $p$-value based on the data from 5).
7. Reject your null hypothesis with error $\alpha$ if $p\leq \alpha$.

See that you have basically decided on everything before you do the experiment.

Now in your case, you would need to decide on $n$ and $\alpha$ beforehand, but also the maximum difference of interest $k$. Since the tests are dependent, you could use the simple Bonferroni correction $\alpha/(k+1)$, which means you have a more conservative significance level for every single one of your tests.