1
$\begingroup$

Two questions:

  1. I have a really hard time with the intuition for why the second part of this statement is true:

    If two statistics have non-overlapping confidence intervals, they are necessarily significantly different, but if they have overlapping confidence intervals, it is not necessarily true that they are not significantly different.

  2. Would it be correct to say that a little overlap doesn't go a long way? Is there a rule of thumb for what degree of overlap implies no difference?

Here's a simple example where looking at overlapping confidence intervals gives the wrong idea. The outcome is a self-reported happiness rating on 7 point Likert scale that I will treat as continuous. The explanatory variables is martial status, gender, age and race. I will use OLS:

. webuse set http://www.stata-press.com/data/ivrm
(prefix now "http://www.stata-press.com/data/ivrm")

. webuse "gss_ivrm.dta", clear

. reg happy7 i.marital i.female i.race c.age

      Source |       SS       df       MS              Number of obs =    1156
-------------+------------------------------           F(  8,  1147) =    7.35
       Model |  53.9791041     8  6.74738801           Prob > F      =  0.0000
    Residual |  1052.29339  1147  .917431026           R-squared     =  0.0488
-------------+------------------------------           Adj R-squared =  0.0422
       Total |  1106.27249  1155  .957811681           Root MSE      =  .95783

--------------------------------------------------------------------------------
        happy7 |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------+----------------------------------------------------------------
       marital |
      widowed  |  -.4531618    .123599    -3.67   0.000    -.6956673   -.2106563
     divorced  |   -.499064   .0795273    -6.28   0.000    -.6550994   -.3430287
    separated  |  -.5496177   .1699695    -3.23   0.001    -.8831036   -.2161317
never married  |   -.155781   .0761826    -2.04   0.041    -.3052539   -.0063082
               |
        female |
       Female  |   .1196351   .0580584     2.06   0.040     .0057225    .2335476
               |
          race |
        black  |  -.0354866   .0850047    -0.42   0.676    -.2022688    .1312956
        other  |  -.1123621   .1179656    -0.95   0.341    -.3438146    .1190905
               |
           age |   .0050064    .002079     2.41   0.016     .0009273    .0090854
         _cons |   5.415511   .1130652    47.90   0.000     5.193673    5.637349
-------------------------------------------------------------------------------

Now I will compute adjusted means for each possible state:

. margins marital

Predictive margins                                Number of obs   =       1156
Model VCE    : OLS

Expression   : Linear prediction, predict()

--------------------------------------------------------------------------------
               |            Delta-method
               |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
---------------+----------------------------------------------------------------
       marital |
      married  |   5.698407   .0408098   139.63   0.000     5.618336    5.778477
      widowed  |   5.245245   .1178389    44.51   0.000     5.014041    5.476449
     divorced  |   5.199342   .0684853    75.92   0.000     5.064972    5.333713
    separated  |   5.148789   .1647873    31.25   0.000     4.825471    5.472107
never married  |   5.542625   .0630837    87.86   0.000     5.418853    5.666398
--------------------------------------------------------------------------------

The CIs for separated and never married overlap by a tiny bit. However, when I do the formal test, I get

. margins ar.marital, contrast(nowald pveffects)

Contrasts of predictive margins
Model VCE    : OLS

Expression   : Linear prediction, predict()

----------------------------------------------------------------------
                              |            Delta-method
                              |   Contrast   Std. Err.      t    P>|t|
------------------------------+---------------------------------------
                      marital |
        (widowed vs married)  |  -.4531618    .123599    -3.67   0.000
       (divorced vs widowed)  |  -.0459022   .1339294    -0.34   0.732
     (separated vs divorced)  |  -.0505536   .1784857    -0.28   0.777
(never married vs separated)  |   .3938366   .1761455     2.24   0.026

The CIs for divorced and windowed overlap by a lot, and the contrast is not significant.

$\endgroup$
  • $\begingroup$ See here for a few references of interest. I've also pointed people towards Using Confidence Intervals for Graphically Based Data Interpretation (Masson & Loftus, 2003) as well. Last time I looked on google scholar there were many similar papers in other fields as well - no economics that I remember though ;). IMO you should upload the pictures of the plots. $\endgroup$ – Andy W Apr 7 '14 at 12:17
  • $\begingroup$ Your first question is answered in several threads here. The second is addressed at stats.stackexchange.com/a/18259. $\endgroup$ – whuber Apr 7 '14 at 15:28