4
$\begingroup$

Here is my situation. I have n predictors of interest, and two control variables.

If I put them all together in a multiple regression, I get issues with colinearity (i.e., VIFs are very high, and the coefficients don't make sense).

It seems that my control variables are causing the colinearity. If I run colineaity diagnositics on just my predictors of interest they seem fine. But when I run the diagnosticts on the predictors of interest AND the control variables, I get high VIF values.

I tried predicting my dependent variable with just my control variables and saving the residuals (in other words residualizing my dependent variable). If I then predict these with my predictors of interest, the results are very interpretable, and I don't have any colineaity issues.

Is this an acceptable way to deal with this?

I'm talking specifically about multiple regression and stepwise regression.

Now I am also conducting a LASSO, to compare results. Is there any harm in also doing the same there?

Long story short. I use my control variables to residualize my dependent variable. I then run a model predicting the residualized dependent variable with my variables of interest. Is that okay?

Improve this question
$\endgroup$
1
  • $\begingroup$ I think I have seen something similar in the works of David Hendry and his students, so this is encouraging. I do not remember what they do first, however: (a) run a regression of the DV on the main variables and then the residuals on the controls (the inverse to what you do) or (b) run a regression of the DV on the controls and then the residuals on the main variables (as you do). $\endgroup$
    – Richard Hardy
    Commented Sep 15, 2020 at 18:30

1 Answer 1

Reset to default
1
+50
$\begingroup$

Fitting the model on the residuals from your control variables is the same as fitting the model all together after orthogonalizing the study variables with respect to controls (it's the same model, but partly rotated). Not the best thing by interpretability standards, but absolutely legit all in all.

Compared to standard OLS, you have two pieces of the same model, the second of which has been rotated to be orthogonal to the first. The covariance structure of $\beta$ will be rotated as well, with the effect of reducing the s. errors as well as the covariances. On the other hand, the estimated effects of the second piece of the model are not the same thing, they are now effects conditioned on the first part of the model (you fit them on the first part residuals, don't you?).

You can also use LASSO, I don't see any reason not to.

edit: I mean apply LASSO on the second part of the model, after running the first one and taking the residuals, that is fine. If you run LASSO on the whole model all together, collinearity is an issue: LASSO tends to select the most effective predictor and bring the other collinear ones to 0, which is generally not ideal. I'll restate it: if you want to do variable selection on the study variables using the residuals of the control variables model, you can do it with LASSO. In any case (LASSO or not) the collinear study variables, to be selected, must show a significant predictive effect in their component that is orthogonal to the control variables. Estimating the model all together, with or without shrinkage, will give a much different result.

Improve this answer
$\endgroup$
9
  • $\begingroup$ There is a well-known reason for not using LASSO here: it tends to fail under multicollinearity. One should use ridge regression instead. $\endgroup$
    – Richard Hardy
    Commented Sep 17, 2020 at 8:13
  • $\begingroup$ @RichardHardy Does using Adaptive LASSO beginning with Ridge Regression help that issue? $\endgroup$
    – Dave
    Commented Sep 17, 2020 at 9:35
  • $\begingroup$ @Dave, no, because LASSO and adaptive LASSO do variable selection while under multicollinearity that is undesirable. There are likely some threads on Cross Validated discussing that. This is also included in textbooks and perhaps even the original LASSO paper (cannot remember properly). $\endgroup$
    – Richard Hardy
    Commented Sep 17, 2020 at 9:49
  • 1
    $\begingroup$ @Dave, I am not sure; this is subtle. $\endgroup$
    – Richard Hardy
    Commented Sep 17, 2020 at 10:43
  • 1
    $\begingroup$ @RichardHardy Not a problem! I didn't mean to load you up with questions. I'll keep searching. Thanks for the help anyway! $\endgroup$
    – Dave
    Commented Sep 17, 2020 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.