I am comparing three group means (95% CI), as following:

A (n=248)          B (n=1000)        C (n=102)
1.3 (0.6-2.1)      3.9 (3.1-4.7)     2.5 (1.2-3.9)

There is statistical significance by ANOVA for overall group difference.

When using Tukey's honesty significant test for pair-wise comparison, I obtain non-significance for A vs. B. Is it possible when they have non-overlapping 95% CI?

  • 1
    $\begingroup$ Are the quoted 95% CI from the ANOVA or from Tukey's HSD? The two use quite different calculations for significance testing, as HSD attempts to correct for the multiple comparisons problem. As one might expect, when one corrects for multiple comparisons, one tends to get more non-significant results than when one doesn't. $\endgroup$ – jbowman Jan 25 '18 at 19:06
  • $\begingroup$ That's derived by the sample mean, sample size, and sd, not from Tukey's test. $\endgroup$ – KubiK888 Jan 25 '18 at 19:39

Yes, it is possible that Tukey's HSD, or any other test procedure for that matter (other than the two-sample T test), will fail to detect significant differences despite the two-sample T test / 95% confidence intervals not overlapping. The converse is also true.

Fundamentally, this is because the formulae involved are different, so you can come up with different results. Along with different formulae often come different assumptions and differences between what is actually being tested (e.g., difference between means vs. difference between medians.)

As an example from a simpler problem, consider two samples of size 10 from Normal distributions with means $(0,1)$ respectively and variances equal to $1$. We compare the two-sample t-test with Mood's test, which tests for differences between medians, but in the case of the Normal distribution, since the median equals the mean, we will use it anyway:

  z <- c(x,y)
  g <- rep(1:2, c(length(x),length(y)))
  m <- median(z)

x1 <- rnorm(10)
x2 <- rnorm(10,1)


If your run this a few times, you will quickly see something like:

    Welch Two Sample t-test

data:  x1 and x2
t = -2.3588, df = 17.91, p-value = 0.0299
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.8988634 -0.1094742
sample estimates:
 mean of x  mean of y 
-0.1981598  0.8060090 

> mood.test(x1,x2)
[1] 0.1788954

where the t-test rejects the null hypothesis (correctly) but the Mood test does not.

The difference between "t-test rejects the null hypothesis" and "95% confidence intervals not overlapping" is merely a matter of degree; the latter implies the former.

In the case of the Tukey HSD: the HSD attempts to correct for the problem of multiple comparisons. Consequently, the confidence intervals for pairwise differences it generates are typically much wider than the confidence intervals generated by pairwise comparisons done without taking the multiple comparisons problem into account. This can easily lead to situations where an ANOVA would reject, but the HSD would fail to reject, a particular pairwise null hypothesis.

As a parenthetical note, the HSD assumes that all of the within-group variances are the same, which is clearly not close to being the case for your data, as it appears the within-group variances for groups A, B, and C respectively are roughly $35$, $160$, and $48$ respectively. You might want to check why that is; if it's a data problem, that might be easily resolvable, but if it appears to be a real effect, it might be having a signficant impact on the HSD.

Here is what happens when we compare the individual confidence intervals with each other and the results of the HSD test on simulated data. The parameters of the simulated data are set to be as close to those I derived from your data above as I could get, given the limited number of digits of accuracy, but that shouldn't matter. I have had mercy on the unwary reader and deleted a lot of irrelevant lines of output. As you will see, in this particular run (the second I performed) the CIs for the means of A and B don't overlap, but the HSD test fails to reject the null hypothesis that the difference between those two means is $0$:

> x <- c(rnorm(248,1.6,5.9), rnorm(1000,3.9,12.65), rnorm(102,2.5,6.9))
> cl <- as.factor(c(rep('A',248), rep('B',1000), rep('C',102)))
> ma <- aov(x~cl)
> TukeyHSD(ma)
  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = x ~ cl)

          diff        lwr       upr     p adj
B-A  1.5447169 -0.2854547 3.3748885 0.1174464
C-A -0.3394765 -3.3741952 2.6952423 0.9627381
C-B -1.8841934 -4.5658370 0.7974502 0.2257234

> t.test(x[cl=='A'])  # Lines not relevant to the CI removed...
95 percent confidence interval:
 1.216763 2.699738

> t.test(x[cl=='B'])  # Lines not relevant to the CI removed

95 percent confidence interval:
 2.743672 4.262262

Given that you are performing multiple comparisons, I'd prefer the Tukey conclusions to the pairwise t-test conclusions, but I'd definitely check that category B data to see why the variance is so much larger than for the other two categories before finalizing my analysis.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.