I have data for two groups, and I am investigating if the two groups perform differently on a task. I have 95% CIs calculated by bootstrapping that give lower and upper bounds to mean point estimates for each group.

I have done a t-test comparing the means and it suggests that the difference between the means is significant, suggesting an effect.

I am having trouble interpreting what the CIs mean, in conjunction with the sig. test. For example, the CIs do overlap to a small degree. Therefore, I interpret that the groups may have similar means that the population level and a significant different may not be observed.

Is this correct?

I am also wondering the best way to report significance tests and CIs as complimentary in a report. Is it alright to discuss a significant difference as important, even though the CIs indicate a degree of overlap?

  • 3
    $\begingroup$ You have received some good answers to your previous questions. Is there a reason you haven't yet accepted any of them? $\endgroup$
    – whuber
    Oct 14 '11 at 16:41

The main message

  • Your hypothesis pertains to differences between two group means. The confidence intervals you mention pertain to the actual group means themselves.
  • You probably want to estimate and report the confidence interval of the difference between group means rather than confidence intervals of the group means.
  • When the 95% confidence interval of the difference between group means does not include zero, then your p-value should also be less than .05 (although exceptions could occur if you used a standard t-test for the significance test and bootstrapping for the confidence intervals). Thus, the tension you discuss between overlapping confidence intervals of individual group means and the significance test will be resolved.

R Example

Here's a little R simulation to demonstrate the idea.

First let us create some data for two groups, n=100 per group, drawn from a population with mean difference between groups of d=.35

> set.seed(4444)
> n <- 100
> g1 <- rnorm(n, mean = 0, sd = 1)
> g2 <- rnorm(n, mean = 0.35, sd = 1)

We can have a quick look at the group means and the difference between group means.

> # Group means
> round(mean(g1), 2)
[1] 0.11
> round(mean(g2), 2)
[1] 0.46
> round(mean(g2) - mean(g1), 2)
[1] 0.35

We can then have a look at the confidence intervals for the individual group means.

> round(as.numeric(t.test(g1, conf.level=.95)$conf.int), 2)
[1] -0.10  0.31
> round(as.numeric(t.test(g2, conf.level=.95)$conf.int), 2)
[1] 0.26 0.66

See how the group means have overlapping confidence intervals (i.e., .31 is larger than .26). However we can look at the confidence interval of interest; i.e., the confidence interval of the difference between group means.

> round(as.numeric(t.test(g2, g1, conf.level=.95)$conf.int), 2)
[1] 0.07 0.64

It does not include zero.

We can now perform at t-test looking at differences between group means and see that the p-value is less than .05 consistent with the confidence interval of the difference between group means.

> t.test(g2, g1)

    Welch Two Sample t-test

data:  g2 and g1 
t = 2.4506, df = 197.308, p-value = 0.01513
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 0.06897783 0.63745786 
sample estimates:
mean of x mean of y 
0.4588689 0.1056511 

95%CI's around your individual group means (I'm assuming it's an independent group design) will overlap when the t-test is significant at 0.05 (they typically don't separate until p<0.01). The minimum distance required between the means is the size of 1 CI bar x sqrt(2). What you'll find if you calculate the CI of the difference between the means (which many t-tests report) is that's it's approximately your original CI x sqrt(2). This is a known method of interpreting those CIs and something you could include in your figure caption if you wish.

To understand this you might recall that the denominator of a t-test has a 2x multiplier for the MSE. That's because the variance of the difference between means is double the variance of the means. The standard deviation is the sqrt of the variance. The standard error used in testing the difference is a standard deviation of the distribution of sampling means. Therefore, the factor differing between individual mean standard errors, and effect standard errors is sqrt(2).

So, that's the way to understand why they can overlap and still be significantly different. For goodness sake though, don't move to standard errors if you find this confusing because they have an even less consistent inferential interpretation that's dependent upon N.

The best way to present your data is with confidence intervals of your means and of your effect. This makes your data easy to interpret for the unitiated and you could even leave the t-test out then. Each of these confidence intervals should be interpreted in your text as including what you believe are likely values for the means and effects. Values outside of these you believe to be unlikely. If 0 is outside your effect CI then you would conclude you had a significant effect. But the CI is much more meaningful than just concluding significance because you are also making a statement about the likely location of the true magnitude of the effect.

  • $\begingroup$ Do you think you could use $\TeX$ markup to make this answer readable, John? $\endgroup$
    – whuber
    Oct 14 '11 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.