Return to Revisions

This is more of comment, but I wanted to include a graph and some code.

I think the statement "if two predictors are correlated and both are included in a model, one will be insignificant" is false if you mean "only one." Binary statistical significance cannot be used for variable selection.

Here's my counterexample using a regression of body fat percentage on thigh circumference, skin fold thickness*, and mid arm circumference:

. webuse bodyfat, clear
(Body Fat)

. reg bodyfat thigh triceps midarm

      Source |       SS       df       MS              Number of obs =      20
-------------+------------------------------           F(  3,    16) =   21.52
       Model |  396.984607     3  132.328202           Prob > F      =  0.0000
    Residual |  98.4049068    16  6.15030667           R-squared     =  0.8014
-------------+------------------------------           Adj R-squared =  0.7641
       Total |  495.389513    19  26.0731323           Root MSE      =    2.48

------------------------------------------------------------------------------
     bodyfat |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       thigh |  -2.856842   2.582015    -1.11   0.285    -8.330468    2.616785
     triceps |   4.334085   3.015511     1.44   0.170    -2.058512    10.72668
      midarm |  -2.186056   1.595499    -1.37   0.190    -5.568362     1.19625
       _cons |   117.0844   99.78238     1.17   0.258    -94.44474    328.6136
------------------------------------------------------------------------------

. corr bodyfat thigh triceps midarm 
(obs=20)

             |  bodyfat    thigh  triceps   midarm
-------------+------------------------------------
     bodyfat |   1.0000
       thigh |   0.8781   1.0000
     triceps |   0.8433   0.9238   1.0000
      midarm |   0.1424   0.0847   0.4578   1.0000


. ellip thigh triceps, coefs plot( (scatteri `=_b[thigh]' `=_b[triceps]'), yline(0, lcolor(gray)) xline(0, lcolor(gray)) legend(off))

As you can see from the regression table, everything is insignificant, though the p-values do vary a bit.

The last Stata command graphs the confidence region for 2 of the regression coefficients (a two dimensional analog of the familiar confidence intervals) along with the point estimates (red dot). The confidence ellipse for the skin fold thickness and thigh circumference coefficients is long, narrow and tilted, reflecting the collinearity in the regressors. There's high negative covariance between the estimated coefficients. The ellipse covers parts of the vertical and the horizontal axes, which means that we cannot reject the individual hypotheses that the $\beta$s are zero, though we can reject the joint null that both are since the ellipse does not cover the origin. In other words, either thigh and triceps are relevant for body fat, but you can't determine which one is the culprit.

So how do we know which predictors would be less significant? The variation in a regressor can be classified into two types:

1. Variation unique to each regressor
2. Variation that is shared by the regressors

In estimating the coefficients of each regressor, only the first will be used. Common variation is ignored since it cannot be allocated, though it is used in prediction and calculating $R^2$. When there is little unique information, the confidence will be low and coefficient variances will be high. The higher the multicollinearity, the smaller the unique variation, and the greater the variances.

*The skin fold is the width of a fold of skin taken over the triceps muscle, and measured using a caliper.