Anova and post-hoc's vs CI's

Question

I'm wondering about the use of anova and post-hoc tests versus CI's. If I have a three-way anova design (age group, sex, strain) with n replicates within each cell, the common way to analyze it would be anova, and then post-hoc tests to look for differences within each grouping factor. What I'm wondering is why can't I look at the confidence intervals instead. If, say, I want to compare the value of young females of one strain to that of young females in the other strain - I can generate the CI for each of the two groups (1.96*SEM for 95%, or using non-parametric bootstrap resampling the individuals with all their info intact) and then look at how much they overlap for a measure of how different they are and how significant are the differences. Seems a lot more straightforward to me then doing all these tests. Would that be valid or am I missing something? Thanks

Answer 1

Comparing means using side-by-side confidence intervals may seem straightforward, but it is in fact inappropriate, so please don’t do it.

Sample means are examples of statistics; and differences between sample means are other statistics. These two kinds of statistics have different sampling distributions and different standard errors — sometimes dramatically different. There are many easy-to-construct examples where CIs overlap substantially but the differences are highly significant. It’s harder to construct an example of the reverse case, but it’s possible.

Repeat: One should not use CIs for means to test differences between them.

Answer 2

Non-technical response: I think if you have a complex model (more complex than a one-way anova, say) especially with unbalanced data, it is more desirable to use post-hoc tests that take into account all the effects in the model, rather than to use confidence intervals for the raw observations which ignore these effects. If you use good software (lsmeans in SAS or emmeans in R), the software makes quick work of multiple comparisons and can even display the output in a compact letter display (cld).

Answer 3

Confidence intervals (CIs) and p values are two ways of looking at the same thing in a frequentist framework: effect size in light of variability.

CIs are built for instance as mean +/- standard error (multiplied by a specific factor corresponding to the distribution of interest).

P values originate for instance from the ratio mean / standard error (interpreted according to the same distribution of interest above).

CIs are always more informative, but p values are more likable (possibly). See for instance this recent paper in JAMA by Chavalarias et al, and the classic yet excellent book by Altman et al.