# Using successive hypothesis tests to test the magnitude of the difference

I don't understand why hypothesis tests only test whether there is a difference in statistics, but not how big the difference is. Is there something wrong with the following approach?

Let $$X_s = \{ X_1, ..., X_n \}$$ and $$Y_s = \{ Y_1, ..., Y_n \}$$ be two sets of samples. Let's say, the sample mean, $$\mu_{X_s}$$, of $$X_s$$ is larger than the sample mean, $$\mu_{Y_s}$$ of $$Y_s$$. I can test the statistical significance of the difference by testing the following hypotheses:

$$H_0\colon \mu_X = \mu_Y \qquad \text{vs.} \qquad H_A\colon \mu_X > \mu_Y.$$

I set the significance level at $$0.05$$, conduct an appropriate test, and get a very small $$p$$-value of, say, $$0.00001$$. I'm now pretty confident that the population means are different.

My results would be stronger if I could show by how much $$\mu_X$$ is bigger than $$\mu_Y$$. Why can't I successively test the following hypotheses

$$H_0^0\colon \mu_X = \mu_Y + 0 \qquad \text{vs.} \qquad H_A^0\colon \mu_X > \mu_Y + 0$$ $$H_0^1\colon \mu_X = \mu_Y + 1 \qquad \text{vs.} \qquad H_A^1\colon \mu_X > \mu_Y + 1$$ $$H_0^2\colon \mu_X = \mu_Y + 2 \qquad \text{vs.} \qquad H_A^2\colon \mu_X > \mu_Y + 2$$

$$\vdots$$

$$H_0^k\colon \mu_X = \mu_Y + k \qquad \text{vs.} \qquad H_A^k\colon \mu_X > \mu_Y + k$$

As long as my $$p$$-value is below $$0.05$$, I could reject the null hypothesis in favor of the alternative, and claim a stronger result, right? For example, if for $$k = 3$$ the $$p$$-value is $$0.046$$ and for $$k = 4$$ it is $$0.061$$ can I not claim that there is statistically significant evidence that $$\mu_X > \mu_Y + 3$$?

Now, I understand that the $$p$$-value is the probability of rejecting the null hypothesis when it is true. So if I conduct $$k > 1$$ independent tests, and always get $$p$$-values less than $$0.05$$, the probability of never rejecting a true null hypothesis is greater than $$0.95^k$$, which approaches zero as $$k$$ grows, so that's a bad idea. But the tests above are clearly not independent and therefore this argument doesn't hold, at least not in this form. What am I missing?

• This is kind of what the confidence interval does. The confidence interval is the range of values that, had they been the null value (e.g. $H_0: \mu = 3$ for a confidence interval of $(2,4)$), the test would not have rejected.
– Dave
Jul 29, 2021 at 21:29

It seems to me that you do not want to use the hypothesis test framework at all. Instead look at the data as evidence and determine the relationship between hypothesised difference sizes and the evidential support.

The procedure you describe is analogous to a little-used 'consonance curve' which is a plot of the hypothesised differences and their p-values. The function is not often (ever?) used, perhaps because the p-values do not have a simple interpretation, but it is worth looking at. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-01105-9

What you should want is a likelihood function that depicts in a straightforwardly interpreted manner the evidential support of the data for the various possible differences.

• This idea is used very often and most recently goes under the name confidence distribution. The p-value shows the plausibility of the hypothesis given the data. Those hypotheses with small p-values are less plausible, those hypotheses with large p-values are more plausible. This is not ascribing a Bayesian interpretation to frequentist inference. It is Fisherian frequentism as opposed to Neyman-Pearson frequentism. Jul 29, 2021 at 22:11

Your suggestion of testing $$k$$ different hypotheses and finding those that are not significant (not rejected) and those that are significant (rejected) is the definition of a confidence interval. See inverting a hypothesis test. See my post here where I invert a likelihood ratio test. This is precisely what is done to construct a Wald confidence interval but it is so deceptively simple that this detail is often overlooked. A confidence interval is a set of plausible true values of $$\mu_X-\mu_Y$$ given the observed data $$\boldsymbol{X}=\boldsymbol{x}$$, $$\boldsymbol{Y}=\boldsymbol{y}$$.

You're running head-first into a big wall of misunderstanding about hypothesis testing :-)

First, you want to do multiple hypothesis tests, so you need to correct your $$p$$-values/significance levels: start reading here.

Second, the hypothesis testing framework is as follows:

2. Translate your question into a null hypothesis/alternative problem.
3. Make assumptions about the distribution of your prospective data and choose the corresponding tests.
4. Decide on the significance level $$\alpha$$ of making an error of the first kind, the power of the test you want to achieve and choose your sample size.
5. Conduct the experiment!
6. Calculate your test statistic and $$p$$-value based on the data from 5).
7. Reject your null hypothesis with error $$\alpha$$ if $$p\leq \alpha$$.

See that you have basically decided on everything before you do the experiment.

Now in your case, you would need to decide on $$n$$ and $$\alpha$$ beforehand, but also the maximum difference of interest $$k$$. Since the tests are dependent, you could use the simple Bonferroni correction $$\alpha/(k+1)$$, which means you have a more conservative significance level for every single one of your tests.

• Thanks a lot for your answer. I’m aware of the problem of testing multiple hypotheses (tried hinting at it with the last paragraph), but can’t see how it applies to the described situation. I don’t want to test different symptoms but rather the magnitude of one symptom. It’s my understanding that hypothesis tests allows one to assess how likely the observed data is, given a particular assumption about the true population, i.e., the null hypothesis. Is this not correct? I’d much rather have a better understanding of the underlying assumptions than blindly follow a recipe. Thanks again!
– red
Apr 11, 2019 at 9:39
• That's kind of the problem with hypothesis testing -- it is a recipe... You're almost right about what hypothesis testing allows you to do: it allows you to assess how likely it is to observe THIS data or even more extreme data GIVEN that your null hypothesis is true. if this probability (the p-value) is very low, it's reasonable to reject the null hypothesis. Apr 11, 2019 at 9:46
• Unfortunately, this doesn't allow you to do tests until you find the "correct difference" in means or something similar. You would need to decide beforehand what difference you want to test and include this difference in the null hypothesis. Apr 11, 2019 at 9:49
• Ok. Do you know of any method that does allow to find the "correct difference" with a certain confidence?
– red
Apr 12, 2019 at 7:59
• No, sorry... Maybe Bayesian statistics could help? BTW, what's your application or are you just asking out of curiosity? Apr 15, 2019 at 7:57