1
$\begingroup$

As the subject suggests, after selecting regressors from lasso regression, is it a good practice to re-run the an ordinary linearly regression with selected variables?

I just feel like intuitively, this would give a more accurate prediction out of sample. Assuming we addressed the redundant variable problems from lasso, then we should just run a normal regression whose whole target is to minimize prediction error, without extra penalty?

$\endgroup$

1 Answer 1

1
$\begingroup$

Such a two-stage estimator makes some sense, but you can do better than that. The research on relaxed lasso (Meinshausen, 2007) suggests that having different penalty intensities for variable selection vs. parameter estimation (for a fixed set of variables) can produce sparser models with equal or better predictive performance than standard lasso. You would use a higher penalty intensity for variable selection and a lower intensity for parameter estimation. Actually, this setup is the dominant approach in the recent extensive study of model selection and regularization in linear models by Hastie et al. (2020). Meanwhile, reducing the penalty intensity all the way to zero (i.e. doing OLS) in the second stage is generally suboptimal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.