# The theory behind Tukey's HSD test.

I have not been able to find satisfactory explanations for the logic behind Tukey's HSD test. However, the resources I have been able to find all concern it's use. It's supposed to make multiple pairwise comparisons of means and identify those pairs which are significantly not equal.

Everything seems to hinge on the Studentized range distribution. Wikipedia says:

Suppose that we take a sample of size n from each of k populations with the same [emphasis mine] normal distribution $$N(\mu,\sigma^2)$$ and suppose that $$y_{\min}$$ is the smallest of these sample means and $$y_{\max}$$ is the largest of these sample means, and suppose $$s^2$$ is the pooled sample variance from these samples. Then the following statistic has a Studentized range distribution.

$$q=\frac{{\overline {y}}_{\max }-{\overline {y}}_{\min }}{s/\sqrt {n}}$$

Accepting this, it would mean that assuming that all groups have identical true variance and mean, we should expect all pairwise mean differences to be smaller (in absolute value) than a certain value (which depends on our desired type 1 error significance level). However, my conclusion from this is that, in practice, if I observe that two groups have significantly different means, then the assumption that all $$k$$ populations have the same mean must be false. I struggle to understand how one can conclude that that specific pair of means is different?

It would make more sense to me if in the result from Wikipedia, the assumptions were relaxed so that each group could have its own mean, and the result still held.

• It you take the null hypothesis (that the observations are all iid $N(\mu, \sigma^2)$, then rejecting the hypothesis because $\frac{{\overline {y}}_{\max }-{\overline {y}}_{\min }}{S\sqrt{2/n}}$ is too big is essentially saying that the particular difference ${\overline {y}}_{\max }-{\overline {y}}_{\min }$ is big enough (above the critical value) to be statistically significant. You might then say the same for any other differences in means above the critical value. Like all tests, this is suggestive rather than conclusive. Commented Mar 26, 2023 at 12:34
• In particular you might find ${\overline {y}}_{a }-{\overline {y}}_{b }$ is not statistically significant and ${\overline {y}}_{b }-{\overline {y}}_{c }$ is not statistically significant but ${\overline {y}}_{a }-{\overline {y}}_{c }$ is statistically significant. This non-transitive result is common: in a exam, you may often say the highest ranking are significantly better than the lowest ranking, without being able to say that the highest are better than those in the middle or that the lowest are worse than those in the middle. Commented Mar 26, 2023 at 12:38
• I understand using this to reject the null hypothesis that all means are equal. My question is: in practice this is used to conclude that specific pairs of means are not equal (ie H0 seems to become: one of these specific pairs identified is in fact equal, and we reject it.) Why? Or what is the exact reasoning? Commented Mar 26, 2023 at 14:47
• That may depend on what you usually say when you reject the null hypothesis (personally I see that a step towards what to look at next). Given the the title of the test is HSD (honestly significantly different), I would report that those sample mean differences which exceed the critical value are significantly different in this test, and stop there. Commented Mar 26, 2023 at 16:00

Here is a visualisation for testing the hypothesis $$\mu_1 = \mu_2 = \mu_3$$ which may be tested based on 3 seperate pairwise comparisons of the means by using individual t-tests based on t statistics that uses a pooled estimate for the deviation.

We simulate the distribution of the outcome of these t-tests for 3 samples of size 20 taken from a standard normal distribution. (See Comparing two, or more, independent paired t-tests) and we plot two of those t-statistics; the other is linearly dependent on the other two, $$t_{1} - t_{2} + t_{3} = 0$$. (Note: the signs in this linear dependency depend on how the t-statistics are computed).

What Tukey's method does, and also Anova, is defining a region such that given the hypothesis, $$\mu_1=\mu_2=\mu_3$$ (and the assumptions of normal distribution and equal variance), the outcome will pass some $$p$$ percent of the time outside the boundary. If the hypothesis is wrong then typically the outcome will be outside the boundary much more often.

Tukey's method uses the boundary based on the maximum magnitude of the t-statistics

$$q = \max(|t_1|,|t_2|,|t_3|)$$

Anova will use a boundary on the overall sizes of the t-statistics

$$F = \frac{t_1^2+t_2^2+t_3^2}{3}$$

Tukey's method will be more sensitive to situations where some $$\mu_i$$ are different in opposite directions (causing a large range). Anova will be more sensitive when several $$\mu_i$$ are different. If only a few $$\mu_i$$ have values at far ends, then Tukey's method will already observe a significant large distance. Anova, will be more sensitive to situations when multiple $$\mu_i$$ cluster at the far ends.

However, my conclusion from this is that, in practice, if I observe that two groups have significantly different means, then the assumption that all k populations have the same mean must be false.

If two means are different then the hypothesis that 'all means are the same' is obviously false, but this doesn't mean that a statement like 'some means are the same' is false. We could have the situation where two means are the same $$\mu_1 = \mu_2$$, and one is different from the others, $$\mu_3 \neq \mu_1$$ and $$\mu_3 \neq \mu_2$$. In such case you will get small t-statistics, except for the ones that compare with the third group. You see this in the graphic where the pointy boundary for Tukey's method extends beyond the boundary for anova when two t-statistics are extreme and the other is nearly zero (Anova is more sensitive for that case).

• Fascinating! Is there a way to show these links to the t statistics, or does one have to go very far into the woods? Commented Apr 3, 2023 at 9:34
• @ThighCrush I am not sure what you mean with 'show these links'? Commented Apr 3, 2023 at 10:26

There are three possibilities

• all the means are the same
• some of the means are the same
• all the means are different

In the third case, you can't make Type I errors, because all the means are truly different, so we can ignore it.

In the first case, the studentised range distribution gives the distribution of the maximum pairwise difference, so the Type I error rate is controlled at exactly the nominal level.

In the second case, consider two means $$\bar X_i$$ and $$\bar X_j$$. If $$\mu_i\neq\mu_j$$ we can't make a Type I error. If $$\mu_i=\mu_j$$ then $$\mu_i$$ and $$\mu_j$$ are two of a set of identical means. The size of this set is less than $$k$$ (by assumption), so the distribution of the studentised range of this set is stochastically smaller than the studentised range of $$k$$ means. The Type I error is controlled conservatively.

This isn't perfect, but it's better than some older methods, which failed to control the Type I error at all in the second case.

• Thank you, this is what I was looking for (I suspected it was something like this). Ideally I would accept both answers! Commented Apr 3, 2023 at 9:19