Two questions:
I have a really hard time with the intuition for why the second part of this statement is true:
If two statistics have non-overlapping confidence intervals, they are necessarily significantly different, but if they have overlapping confidence intervals, it is not necessarily true that they are not significantly different.
- Would it be correct to say that a little overlap doesn't go a long way? Is there a rule of thumb for what degree of overlap implies no difference?
Here's a simple example where looking at overlapping confidence intervals gives the wrong idea. The outcome is a self-reported happiness rating on 7 point Likert scale that I will treat as continuous. The explanatory variables is martial status, gender, age and race. I will use OLS:
. webuse set http://www.stata-press.com/data/ivrm
(prefix now "http://www.stata-press.com/data/ivrm")
. webuse "gss_ivrm.dta", clear
. reg happy7 i.marital i.female i.race c.age
Source | SS df MS Number of obs = 1156
-------------+------------------------------ F( 8, 1147) = 7.35
Model | 53.9791041 8 6.74738801 Prob > F = 0.0000
Residual | 1052.29339 1147 .917431026 R-squared = 0.0488
-------------+------------------------------ Adj R-squared = 0.0422
Total | 1106.27249 1155 .957811681 Root MSE = .95783
--------------------------------------------------------------------------------
happy7 | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
marital |
widowed | -.4531618 .123599 -3.67 0.000 -.6956673 -.2106563
divorced | -.499064 .0795273 -6.28 0.000 -.6550994 -.3430287
separated | -.5496177 .1699695 -3.23 0.001 -.8831036 -.2161317
never married | -.155781 .0761826 -2.04 0.041 -.3052539 -.0063082
|
female |
Female | .1196351 .0580584 2.06 0.040 .0057225 .2335476
|
race |
black | -.0354866 .0850047 -0.42 0.676 -.2022688 .1312956
other | -.1123621 .1179656 -0.95 0.341 -.3438146 .1190905
|
age | .0050064 .002079 2.41 0.016 .0009273 .0090854
_cons | 5.415511 .1130652 47.90 0.000 5.193673 5.637349
-------------------------------------------------------------------------------
Now I will compute adjusted means for each possible state:
. margins marital
Predictive margins Number of obs = 1156
Model VCE : OLS
Expression : Linear prediction, predict()
--------------------------------------------------------------------------------
| Delta-method
| Margin Std. Err. t P>|t| [95% Conf. Interval]
---------------+----------------------------------------------------------------
marital |
married | 5.698407 .0408098 139.63 0.000 5.618336 5.778477
widowed | 5.245245 .1178389 44.51 0.000 5.014041 5.476449
divorced | 5.199342 .0684853 75.92 0.000 5.064972 5.333713
separated | 5.148789 .1647873 31.25 0.000 4.825471 5.472107
never married | 5.542625 .0630837 87.86 0.000 5.418853 5.666398
--------------------------------------------------------------------------------
The CIs for separated and never married overlap by a tiny bit. However, when I do the formal test, I get
. margins ar.marital, contrast(nowald pveffects)
Contrasts of predictive margins
Model VCE : OLS
Expression : Linear prediction, predict()
----------------------------------------------------------------------
| Delta-method
| Contrast Std. Err. t P>|t|
------------------------------+---------------------------------------
marital |
(widowed vs married) | -.4531618 .123599 -3.67 0.000
(divorced vs widowed) | -.0459022 .1339294 -0.34 0.732
(separated vs divorced) | -.0505536 .1784857 -0.28 0.777
(never married vs separated) | .3938366 .1761455 2.24 0.026
The CIs for divorced and windowed overlap by a lot, and the contrast is not significant.
