Help needed with wording: significant effect size but overlapping CIs I have an issue with describing the results of a simple analysis in a self-consistent way.
This is for a study of an intervention that took place in 25 different towns, to reduce the number of cot deaths that took place. Data is for the number of cot deaths before the intervention started (the 'baseline' phase), and for the number after the intervention was rolled out. Both baseline and intervention phases went for a year.
I've done a Poisson regression for the number of cot deaths, adjusting for clustering at the town level. I get a significant effect (reduction by 20%, with 95% C.I 9 - 30.2), but the issues arise when I try to state the estimated rate of cot death before and after the intervention.
Adjusted again for clustering at the town level, I get 113 per 100,000 (95% CI 96 - 130) at baseline, and 92 per 100,000 (95% CI 75 - 120) after the intervention. So although the effect size is significant, the confidence intervals for the rates overlap!
I'm assuming that comparing the confidence intervals on the rates would be misleading because they arise from marginal distributions and do not reflect the pairing between towns from the baseline to intervention phase. But how should I describe these results to avoid this misleading conclusion?
 A: Some overlap can occur in CIs with a significant effect but yours is too much. You are correct that the problem you're seeing is that the CI of the absolute measurements isn't taking into account the pairing.
When you design a study that's paired or repeated measures then you're usually designing it to measure the effect. Typically, as a consequence of the improved sensitivity from the pairing, the N isn't large and you don't have a narrow range in the estimate of the separate group values. Therefore, given that the only thing you attempted to measure well was the effect then that's the CI you report. Don't bother reporting the CI of the separate conditions.
To avoid confusion in unsophisticated readers, but still report your data, report the standard deviation of the raw values but downplay them (e.g. an appendix with full results). Anyone who wants to derive the CI can then do so as long as you've included the N. It's also a good idea to report the correlation across the pairing. That way your results can be more easily simulated and meta-analyzed.
