1
$\begingroup$

I am not sure if I've made a computational error, am failing to think something through (too late in the day), or just really don't understand confidence intervals as much as I was hoping I did, but I ran a linear model with Condition as a predictor (2 groups) and the slope representing the group difference was significantly different from zero:

b = 0.10599, t = 2.043, p = 0.045, 95% CIs [0.0023, 0.21]

However, when I compute the CIs for the two group means they overlap:

Experimental: M = .32, 95% CIs [.24, .39]

Control: M = .21, 95% CIs [.14, .28].  

What's going on? The overlapping CIs suggest to me that the means aren't significantly different, but I thought testing whether the mean difference differs significantly from zero is the same as testing whether the means differ significantly from each other?

$\endgroup$
5
$\begingroup$

See this page for an explanation of the relation between overlapping confidence intervals and significance tests for the difference between 2 groups. No overlap of 95% confidence intervals is much too stringent a test for the difference between the two means at p = 0.05. Each mean value would be in the space outside its own 95% CI only a fraction of 0.05 of the time in repeated samplings. Crudely, requiring lack of overlap of the two 95% might be thought of as requiring this restriction to hold simultaneously for both mean values, which would occur only (0.05)^2 = 0.0025 fraction of the time. More detailed calculations on the linked page show that non-overlap of 95% CIs is about equivalent to a p = 0.005 test for the difference between the 2 means, under reasonable assumptions.

$\endgroup$
  • $\begingroup$ Thanks very much! I'll bet that this is one reason why people tend to use SE bars instead of CIs for bar charts! $\endgroup$ – PanPsych Mar 26 '16 at 13:26
  • 1
    $\begingroup$ Those who tend to use bar charts a lot probably haven't thought much about what error bars to display, other than that SEs are smaller. Showing non-overlap of 83% CI would approximately be equivalent to p = 0.05 for a difference in means according to the page I linked in my answer, but I'm unaware of any groundswell of support for such displays. $\endgroup$ – EdM Mar 26 '16 at 13:49
  • $\begingroup$ I've heard Geoff Cummings talk about shifting towards using CIs instead of null hypothesis testing, and I believe he and others have argued that CIs should be displayed in bar charts instead of SEs. I would do this if there was a good reason to, but I worry that others may get confused as I did. (I could avoid the bar chart altogether but for some reason people like to see them.) $\endgroup$ – PanPsych Mar 26 '16 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.