1
$\begingroup$

I'm working on a regression model with the Hitters data from the ISLR package.

It has ~300 observations and 20 variables. I want to predict a player's salary.

I have major problems of collinearity and multicollinearity.

Initialize data

library('ISLR')
data(Hitters)
attach(Hitters)

Check for multicollinearity

library('mctest')    
imcdiag(num_vars(Hitters[-19]),Hitters$Salary)

Examine Output

Call:
imcdiag(x = num_vars(Hitters[-19]), y = Hitters$Salary)


All Individual Multicollinearity Diagnostics Result

             VIF    TOL        Wi        Fi Leamer      CVIF Klein
AtBat    22.4794 0.0445  353.6936  380.4917 0.2109   -5.2419     1
Hits     30.0835 0.0332  478.9086  515.1937 0.1823   -7.0152     1
HmRun     7.6367 0.1309  109.2844  117.5645 0.3619   -1.7808     1
Runs     15.1175 0.0661  232.4681  250.0814 0.2572   -3.5252     1
RBI      11.6885 0.0856  176.0040  189.3392 0.2925   -2.7256     1
Walks     4.0903 0.2445   50.8876   54.7431 0.4944   -0.9538     1
Years     9.1263 0.1096  133.8123  143.9508 0.3310   -2.1281     1
CAtBat  250.0646 0.0040 4101.2642 4412.0019 0.0632  -58.3124     1
CHits   495.6521 0.0020 8145.2706 8762.4079 0.0449 -115.5807     1
CHmRun   46.2836 0.0216  745.6703  802.1670 0.1470  -10.7928     1
CRuns   158.6813 0.0063 2596.4848 2793.2109 0.0794  -37.0028     1
CRBI    131.2023 0.0076 2143.9984 2306.4413 0.0873  -30.5950     1
CWalks   19.7303 0.0507  308.4253  331.7935 0.2251   -4.6009     1
PutOuts   1.2304 0.8128    3.7936    4.0810 0.9015   -0.2869     0
Assists   2.7002 0.3703   27.9969   30.1181 0.6086   -0.6297     1
Errors    2.1842 0.4578   19.5002   20.9777 0.6766   -0.5093     1

1 --> COLLINEARITY is detected by the test 
0 --> COLLINEARITY is not detected by the test

HmRun , Runs , RBI , Years , CAtBat , CHits , CHmRun , CRBI , Assists , Errors , coefficient(s) are non-significant may be due to multicollinearity

R-square of y on all x: 0.5279

View correlation matrix

enter image description here

I see three areas for problematic collinearity. The runs/hits/hmruns/rbi/walks group in the bottom left of the matrix, the years/catbat/chits/so on in the middle of the matrix and the error:assist pair in the top right of the matrix.

I know my options include: drop one of the variables involved, change them to an interaction term.

The problem is that it seems like I'm just guessing. I have no reason to think error better describes what's going on than assist and it is too many potential combinations to try them all.

What's the better alternative for dealing with collinearity?

|cite|improve this question
$\endgroup$
  • 4
    $\begingroup$ Could you explain what is "problematic" about collinearity, given your objective is only prediction? $\endgroup$ – whuber♦ Jan 18 '19 at 20:24
  • $\begingroup$ My understanding is that collinearity creates unreliable regression coefficients because it can attribute that association to either variable. Multicollinearity is the case where the relationship is between a group of one or more variables and another group of one or more variables. $\endgroup$ – Sebastian Jan 18 '19 at 20:26
  • 4
    $\begingroup$ Your objective is not to "attribute association" to variables: as you stated it, it is solely prediction. That is unaffected by collinearity. $\endgroup$ – whuber♦ Jan 18 '19 at 20:28
  • $\begingroup$ Makes sense. Perhaps this belongs in a separate post, I'm curious how to deal with a case like this when my goal is explanation. $\endgroup$ – Sebastian Jan 18 '19 at 20:36
  • 3
    $\begingroup$ For answers to that, please search our site for threads on "model selection" and even "overfitting." You will discover this is a huge topic with many approaches. $\endgroup$ – whuber♦ Jan 18 '19 at 20:40
4
$\begingroup$

You don't really want stats like "career at-bats" and "years" in the same model with counting stats like "career hits" and so forth. Before even looking at the data, those seem an obvious situation for computing rate statistics (hits per at-bat, etc.).

Once you tuck "career at-bats" into the denominator of a bunch of rate stats, you can then may be able to use "years" to represent career longevity or something (but think carefully about whether that's something you have a substantive reason to believe matters).

So my advice is to convert all the one-year stats to rates using AtBat as denominator and the career stats to rates using CAtBat as denominator. Then use the rate stats for modeling. Either current-year and career models separately or perhaps, if the data supports it and your a priori assumptions require it, combine them together in one model.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ That is a better way to compare players. Doesn't that still leave collinearity? If I have a high CHits / CAtBat rate, I probably also have a high CHmRun / CAtBat rate. $\endgroup$ – Sebastian Jan 18 '19 at 19:03
  • 1
    $\begingroup$ At that point it become IMO an empirical question. No doubt some sort of correlation exists between hits and HR's (rates) but necessarily to the level we term "collinear". Nothing at all wrong with IV's in a model having modest correlations, the problem arises when one IV (or a group of IV's together) become near-duplicates of some other term in the model. My intuition says that there's plenty of guys in the dataset who get a lot of hits (per at-bat) without getting a lot of home runs but as I say, it's more of a question to answer with the data rather than reasoning from first principles. $\endgroup$ – Brent Hutto Jan 18 '19 at 19:29
  • $\begingroup$ Can you explain why it's important that all IVs have the same denominator? This is new to me. $\endgroup$ – Parseltongue Jan 18 '19 at 20:04
  • $\begingroup$ There is no rule about "same denominator". The principle is that some quantities are essentially rates. When you split the numerator and denominator apart, those separate quantities do not coexist well in regression models. You can sometimes get acceptable results by modeling "salary predicted by number of hits, adjusting for number of at-bats" but more often you'll get better estimates from a model of "salary predicted by number of hits per at-bat". $\endgroup$ – Brent Hutto Jan 18 '19 at 20:14
1
$\begingroup$

If your only objective is prediction and you don't care about interpreting your model coefficients, then you don't need to worry at all about multicollinearity. You simply use an out-of-sample dataset to test your model (with multicollinearity or not) and so long as it produces the smallest mean-squared predicted error (or your other favorite prediction statistic), you are all set -- you don't have to wring your hands over multicollinearity. Multicollinearity only affects the direction, magnitude, and stability of coefficients, which, if you aren't using them for interpretation, should not concern you.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Does the same apply to interaction terms- their usefulness is interpretation, not predictive value? $\endgroup$ – Sebastian Jan 18 '19 at 20:10
  • 1
    $\begingroup$ Correct. If all you are concerned with is predictability, then you need not be concerned with even examining significance really or interpretation. All you should be concerned with is minimizing prediction error. $\endgroup$ – StatsStudent Jan 18 '19 at 20:22
  • 4
    $\begingroup$ Not necessarily--after all, the response by Brent Hutto is not algorithmic: it's based on thinking about what the predictor variables mean and how they might be related to the response. No matter what prediction procedure you might select, nothing exists that can automatically (or algorithmically) replace that knowledge. $\endgroup$ – whuber♦ Jan 18 '19 at 20:26
  • 1
    $\begingroup$ I still do not understand what you mean exactly. Stating that multicolinearity changes the magnitude of estimated coefficients, what do you mean by this? Mulicollinearity from my understanding leads to very small eigenvalues of (x'x) which inevitably leads to an inverse with particularly large entries. Thus, estimated coefficients become inflated. And model instability, which I interpret as small changes in data leading to large changes in predictions, has no affect on predictive performance? $\endgroup$ – aranglol Jan 19 '19 at 22:06
  • 1
    $\begingroup$ This also seens to contradict a passage I read in Kuhn's/Johnson's Applied Predictive Modelling, in particularly page 45-46 in which they claim that "in (unregularized) linear regression, using highly correlated predictors can result in highly unstable models, numerical issues, and degraded predictive performance." $\endgroup$ – aranglol Jan 19 '19 at 22:23
1
$\begingroup$

This response addresses concerns with collinearity when the goal is description and inference, not prediction.

In linear models, some level of collinearity will always be present in data where the presence of strong collinearity suggests predictors that are redundant with one another. It then becomes a matter of choice as to which predictors to retain and which to eliminate as useless, uninformative or less informative. At that point the questions are: how much collinearity can be present before a regression model's performance degrades? What are some approaches to managing the problem?

Usually this isn't a simple, one-step solution. Rather, it's step-by-step.

Cookbook rules-of-thumb for VIFs abound but the goal is always to minimize the risk of model degradation from collinearity. One prescription is for absolute VIF values around 5 where VIFs greater than 5 are considered problematic and VIFs less than five acceptable or normal (different sources prescribe different cut-offs).

Since collinearity is a function of undesirable dependence among predictors, analyzing a matrix of pairwise Pearson correlations is not only not helpful, it can actually be misleading. First, the correlations are pairwise. This means they don't reflect association with other predictors. Moreover, the magnitude of the correlations is not a useful diagnostic wrt any actual underlying collinearity as lower valued correlations can be collinear while big magnitude correlations may not be.

A better first step would be exploratory: start with a matrix of pairwise correlations and scatterplots between the dependent variable and the predictors, taken one at a time. This is useful in identifying predictors that have little or no association with the DV. Chances are that if they can't be transformed into useful predictors, then they can be eliminated as uninformative. Retain both the magnitude and sign (pos./neg.) of these DV-predictor correlations as these metrics can be useful in later steps.

Assuming you have identified a reduced set of variates, a much better diagnostic than pairwise correlations would be to examine a matrix of partial correlations (excluding the dependent variable). You are looking for high magnitude, absolute valued partials. This should target problematic, pairwise dependence between predictors.

Factor analysis with rotation can also be useful. First, standardize the variables (again, excluding the dependent variable) to a mean of 0 and std deviation of 1 and then run a factor analysis with rotation. Based on the factor loadings you will gain insight into dependence among predictors. Some analysts even use the factor loadings as a decision tool wrt variates to retain vs eliminate. Simply retain those variables with the highest loadings on each factor and throw away the rest. (Note that throwing away potentially useful variables that are not top-loading can be wasteful.)

Having a reduced set of variates does not mean that collinearity has been eliminated. The next step is to run the regression on this reduced set being sure to include coefficients, std errors, t-values along with VIFs, tolerances, etc., as you have reported in your post. The VIF values will have changed relative to those reported in your original post above.

So, having identified a set of problematic variables based on their partial correlations, a factor analysis and/or the new VIFs, the issue becomes one of managing the collinearity. Again, this isn't a simple, one-step process. In doing so you don't want to be too clinical and reduce collinearity down to zero.

Start with the largest magnitude VIF. Identify that variate's collinear predictor (VIFs are uninformative about this) based on the diagnostics from the partials or factor structure and, based on the results of the regression, look at the coefficients and absolute t-values for this pair. Usually one of the coefficients is wrong-signed wrt the original correlation between the DV and predictors (from the step above). In addition, the absolute t-value for this wrong-signed predictor is usually smaller. At this point it's a safe decision to eliminate this wrong-signed, smaller t-value predictor and retain the correctly signed predictor.

Rerun the regression on this new set of reduced variates, repeating the inspection for the next largest pair of VIFs. Iterate until the metrics reach a level of collinearity you are comfortable with, e.g., reduced models with VIFs around some preferred cutoff or threshold. Always bear in mind that cookbook rules such as VIF cut-offs can be bent, if not broken.

In this way you should be able to build a reduced model that minimizes any risk of degradation as a function of collinearity.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

I would identify the 9 predictors which are giving You the most difficulty. Then generate All possible models associated with deleting Or keeping the 9 predictors. So that will Give you $2^9=512$ models. Then eliminate Those models which exhibit multicollinearity. For the remaining models evaluate the takeuchi Information criterion for each model. The takeuchi information criterion is also Called the generalized akaike information criterion. Don't use aic because some models may be misspecified. Then choose model with smallest takeuchi information Criterion from remaining models.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Is TIC a better diagnostic for choosing the model than cross validation? $\endgroup$ – Sebastian Jan 18 '19 at 20:07
  • $\begingroup$ TIC is approximately equivalent to cross-validation (e.g., see Model Selection and Model Averaging book by Claeskens and Hjort). Convergence rate for TIC is slightly worse than cross-validation. Both TIC and cross-validation assume a locally unique solution. An advantage of TIC over cross-validation, however, is that the computation is a simple formula so it is computed without having to do excessive simulations. If the data set is small, either cross-validation or TIC could be used but if excessive computation is required for parameter estimation then TIC will compute answer instantly. $\endgroup$ – RMG Jan 20 '19 at 1:59
0
$\begingroup$

The problem of fitting a linear model when there are collinear predictors is that the coefficients are unstable among the correlated variables because all of them can be used to explain the same variance. One way to deal with this problem is to add regularization. Lasso will discard most of the correlated predictors. I'd say elastic net and ridge regression are also useful for this but cannot remember exactly how.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.