I am using R to perform a linear regression with a dataset that has clearly correlated independent variables (collinearity). I am using the vif (variance inflation factor) function from the car package to quantify and examine collinearity. I would like to eliminate some independent variables from the model in order to reduce collinearity in a systematic way and I am currently using a glmnet (elastic-net) model via the caret package to do this.

However, the optimal cross validated model doesn't quite eliminate some problematic correlated independent variables from the model. I would like to force the model to set the coefficients for some of these variables all the way to 0 by increasing the penalization factor (lambda) of the glmnet model.

Is there a good way to accomplish this in an elegant way while allowing my model to optimize on both the alpha and lambda hyperparameters? My current thought is to take the alpha and lambda values from the 'optimal' glmnet model that caret produces and then directly use the glmnet package with these hyperparameter values, increasing lambda until I am satisfied with the level of collinearity in the model. Would this be a reasonable approach?

  • 1
    $\begingroup$ If you want some of the variables to be eliminated from the model, used lasso. If you want the variables to remain, use ridge regression. I would question the idea of eliminating some of the highly correlated variables; this is often considered unsatisfactory. This is also basically the argument for why ridge (or elastic net) should be used instead of lasso when highly correlated regressors are present. Your thinking seems to be going in the opposite direction. $\endgroup$ – Richard Hardy Sep 27 '18 at 16:26
  • $\begingroup$ To clarify, I could use LASSO (elastic-net model with the alpha fixed at 1) for variable selection, but my thinking is that it would be more ideal to allow for an optimal alpha to be selected. For the sake of creating an interpretable model, eliminating variables using elastic-net or LASSO is an attractive option in my use case. I understand if I were more focused on performance rather than interpretability, leaving all the variables present and using ridge would probably result in a better fitting model. $\endgroup$ – cambonator Sep 27 '18 at 16:50
  • $\begingroup$ What is the goal of your model? Prediction? If so, examine a grid of both alpha and lambda values. (When I was using glmnet 3-4 years ago, there was an option to select an optimal lambda but not alpha, so I had to do alpha selection myself on a manually created grid.) Use cross-validation to select the values that yield the lowest cross-validation error (absolute error, squared error – choose the one that is relevant in your application). $\endgroup$ – Richard Hardy Sep 27 '18 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.