How do I handle terms with collinearity?

Question

I'm working on a regression model with the Hitters data from the ISLR package.

It has ~300 observations and 20 variables. I want to predict a player's salary.

I have major problems of collinearity and multicollinearity.

Initialize data

library('ISLR')
data(Hitters)
attach(Hitters)

Check for multicollinearity

library('mctest')    
imcdiag(num_vars(Hitters[-19]),Hitters$Salary)

Examine Output

Call:
imcdiag(x = num_vars(Hitters[-19]), y = Hitters$Salary)


All Individual Multicollinearity Diagnostics Result

             VIF    TOL        Wi        Fi Leamer      CVIF Klein
AtBat    22.4794 0.0445  353.6936  380.4917 0.2109   -5.2419     1
Hits     30.0835 0.0332  478.9086  515.1937 0.1823   -7.0152     1
HmRun     7.6367 0.1309  109.2844  117.5645 0.3619   -1.7808     1
Runs     15.1175 0.0661  232.4681  250.0814 0.2572   -3.5252     1
RBI      11.6885 0.0856  176.0040  189.3392 0.2925   -2.7256     1
Walks     4.0903 0.2445   50.8876   54.7431 0.4944   -0.9538     1
Years     9.1263 0.1096  133.8123  143.9508 0.3310   -2.1281     1
CAtBat  250.0646 0.0040 4101.2642 4412.0019 0.0632  -58.3124     1
CHits   495.6521 0.0020 8145.2706 8762.4079 0.0449 -115.5807     1
CHmRun   46.2836 0.0216  745.6703  802.1670 0.1470  -10.7928     1
CRuns   158.6813 0.0063 2596.4848 2793.2109 0.0794  -37.0028     1
CRBI    131.2023 0.0076 2143.9984 2306.4413 0.0873  -30.5950     1
CWalks   19.7303 0.0507  308.4253  331.7935 0.2251   -4.6009     1
PutOuts   1.2304 0.8128    3.7936    4.0810 0.9015   -0.2869     0
Assists   2.7002 0.3703   27.9969   30.1181 0.6086   -0.6297     1
Errors    2.1842 0.4578   19.5002   20.9777 0.6766   -0.5093     1

1 --> COLLINEARITY is detected by the test 
0 --> COLLINEARITY is not detected by the test

HmRun , Runs , RBI , Years , CAtBat , CHits , CHmRun , CRBI , Assists , Errors , coefficient(s) are non-significant may be due to multicollinearity

R-square of y on all x: 0.5279 

View correlation matrix

I see three areas for problematic collinearity. The runs/hits/hmruns/rbi/walks group in the bottom left of the matrix, the years/catbat/chits/so on in the middle of the matrix and the error:assist pair in the top right of the matrix.

I know my options include: drop one of the variables involved, change them to an interaction term.

The problem is that it seems like I'm just guessing. I have no reason to think error better describes what's going on than assist and it is too many potential combinations to try them all.

What's the better alternative for dealing with collinearity?

Answer 1

You don't really want stats like "career at-bats" and "years" in the same model with counting stats like "career hits" and so forth. Before even looking at the data, those seem an obvious situation for computing rate statistics (hits per at-bat, etc.).

Once you tuck "career at-bats" into the denominator of a bunch of rate stats, you can then may be able to use "years" to represent career longevity or something (but think carefully about whether that's something you have a substantive reason to believe matters).

So my advice is to convert all the one-year stats to rates using AtBat as denominator and the career stats to rates using CAtBat as denominator. Then use the rate stats for modeling. Either current-year and career models separately or perhaps, if the data supports it and your a priori assumptions require it, combine them together in one model.

Answer 2

If your only objective is prediction and you don't care about interpreting your model coefficients, then you don't need to worry at all about multicollinearity. You simply use an out-of-sample dataset to test your model (with multicollinearity or not) and so long as it produces the smallest mean-squared predicted error (or your other favorite prediction statistic), you are all set -- you don't have to wring your hands over multicollinearity. Multicollinearity only affects the direction, magnitude, and stability of coefficients, which, if you aren't using them for interpretation, should not concern you.

Answer 3

This response addresses concerns with collinearity when the goal is description and inference, not prediction.

In linear models, some level of collinearity will always be present in data where the presence of strong collinearity suggests predictors that are redundant with one another. It then becomes a matter of choice as to which predictors to retain and which to eliminate as useless, uninformative or less informative. At that point the questions are: how much collinearity can be present before a regression model's performance degrades? What are some approaches to managing the problem?

Usually this isn't a simple, one-step solution. Rather, it's step-by-step.

Cookbook rules-of-thumb for VIFs abound but the goal is always to minimize the risk of model degradation from collinearity. One prescription is for absolute VIF values around 5 where VIFs greater than 5 are considered problematic and VIFs less than five acceptable or normal (different sources prescribe different cut-offs).

Since collinearity is a function of undesirable dependence among predictors, analyzing a matrix of pairwise Pearson correlations is not only not helpful, it can actually be misleading. First, the correlations are pairwise. This means they don't reflect association with other predictors. Moreover, the magnitude of the correlations is not a useful diagnostic wrt any actual underlying collinearity as lower valued correlations can be collinear while big magnitude correlations may not be.

A better first step would be exploratory: start with a matrix of pairwise correlations and scatterplots between the dependent variable and the predictors, taken one at a time. This is useful in identifying predictors that have little or no association with the DV. Chances are that if they can't be transformed into useful predictors, then they can be eliminated as uninformative. Retain both the magnitude and sign (pos./neg.) of these DV-predictor correlations as these metrics can be useful in later steps.

Assuming you have identified a reduced set of variates, a much better diagnostic than pairwise correlations would be to examine a matrix of partial correlations (excluding the dependent variable). You are looking for high magnitude, absolute valued partials. This should target problematic, pairwise dependence between predictors.

Factor analysis with rotation can also be useful. First, standardize the variables (again, excluding the dependent variable) to a mean of 0 and std deviation of 1 and then run a factor analysis with rotation. Based on the factor loadings you will gain insight into dependence among predictors. Some analysts even use the factor loadings as a decision tool wrt variates to retain vs eliminate. Simply retain those variables with the highest loadings on each factor and throw away the rest. (Note that throwing away potentially useful variables that are not top-loading can be wasteful.)

Having a reduced set of variates does not mean that collinearity has been eliminated. The next step is to run the regression on this reduced set being sure to include coefficients, std errors, t-values along with VIFs, tolerances, etc., as you have reported in your post. The VIF values will have changed relative to those reported in your original post above.

So, having identified a set of problematic variables based on their partial correlations, a factor analysis and/or the new VIFs, the issue becomes one of managing the collinearity. Again, this isn't a simple, one-step process. In doing so you don't want to be too clinical and reduce collinearity down to zero.

Start with the largest magnitude VIF. Identify that variate's collinear predictor (VIFs are uninformative about this) based on the diagnostics from the partials or factor structure and, based on the results of the regression, look at the coefficients and absolute t-values for this pair. Usually one of the coefficients is wrong-signed wrt the original correlation between the DV and predictors (from the step above). In addition, the absolute t-value for this wrong-signed predictor is usually smaller. At this point it's a safe decision to eliminate this wrong-signed, smaller t-value predictor and retain the correctly signed predictor.

Rerun the regression on this new set of reduced variates, repeating the inspection for the next largest pair of VIFs. Iterate until the metrics reach a level of collinearity you are comfortable with, e.g., reduced models with VIFs around some preferred cutoff or threshold. Always bear in mind that cookbook rules such as VIF cut-offs can be bent, if not broken.

In this way you should be able to build a reduced model that minimizes any risk of degradation as a function of collinearity.

Answer 4

I would identify the 9 predictors which are giving You the most difficulty. Then generate All possible models associated with deleting Or keeping the 9 predictors. So that will Give you $2^9=512$ models. Then eliminate Those models which exhibit multicollinearity. For the remaining models evaluate the takeuchi Information criterion for each model. The takeuchi information criterion is also Called the generalized akaike information criterion. Don't use aic because some models may be misspecified. Then choose model with smallest takeuchi information Criterion from remaining models.

Answer 5

The problem of fitting a linear model when there are collinear predictors is that the coefficients are unstable among the correlated variables because all of them can be used to explain the same variance. One way to deal with this problem is to add regularization. Lasso will discard most of the correlated predictors. I'd say elastic net and ridge regression are also useful for this but cannot remember exactly how.