Recently I have found that in the applied econometrics literature, when dealing with feature selection problems, it is not uncommon to perform LASSO followed by an OLS regression using the selected variables.

I was wondering how can we qualify the validness of such a procedure. Will it cause troubles such as omitted variables? Any proofs showing that it is more efficient, or the results are more interpretable?

Here are some related discussions:

Variable selection with LASSO

Using trees after variable selection using Lasso/Random

If, as pointed out, such a procedure is not correct in general, then why are still there so many researches doing so? Can I say that it is just a rule of thumb, a compromise solution, due to some of the uneasy properties of LASSO estimator, and people's fondness towards OLS?

  • $\begingroup$ Could you explain what it means to do "OLS regression" after performing the LASSO? What, specifically, is this OLS step attempting to estimate that the LASSO has not estimated? $\endgroup$ – whuber Dec 29 '16 at 2:03
  • 2
    $\begingroup$ There are a few recent working papers on the subject. Many seem to require the assumption that the set of valid variables is sparse. If that assumption doesn't hold, then yes omitted variables bias would be present. And people like ols because they want to interpret coefs as unbiased out of sample marginal effects. Econometrics is fairly stuck in that paradigm. $\endgroup$ – generic_user Dec 29 '16 at 2:38
  • 4
    $\begingroup$ In this recent LASSO book (free online), section 11.4 appears to address this issue. I have not read this in detail, but the introduction ends by saying "Given [a LASSO estimate] $\hat{\beta}$ that correctly recovers the support of $\beta^*$, we can estimate $\beta^*$ very well ... simply by performing an ordinary least-squares regression restricted to this subset." $\endgroup$ – GeoMatt22 Dec 29 '16 at 4:04

There was a similar question a few days ago which had the relevant reference:

  • Belloni, A., Chernozhukov, V., and Hansen, C. (2014) "Inference on Treatment Effects after Selection among High-Dimensional Controls", Review of Economic Studies, 81(2), pp. 608-50 (link)

At least for me the paper is a pretty tough read because the proofs behind this relatively simple are fairly elaborate. When you are interested in estimating a model like $$y_i = \alpha T_i + X_i'\beta + \epsilon_i$$

where $y_i$ is your outcome, $T_i$ is some treatment effect of interest, and $X_i$ is a vector of potential controls. The target parameter is $\alpha$. Assuming that most of the variation in your outcome is explained by the treatment and a sparse set of controls, Belloni et al. (2014) develop a double-robust selection method which provides correct point estimates and valid confidence intervals. This sparsity assumption is important though.

If $X_i$ includes a few important predictors of $y_i$ but you don't know which they are (either single variables, their higher order polynomials, or interactions with other variables), you can perform a three step selection procedure:

  1. regress $y_i$ on $X_i$, their squares, and interactions, and select important predictors using LASSO
  2. regress $T_i$ on $X_i$, their squares, and interactions, and select important predictors using LASSO
  3. regress $y_i$ on $T_i$ and all the variables which were selected in either of the first two steps

They provide proofs as for why this works and why you get the correct confidence intervals etc out of this method. They also show that if you only perform a LASSO selection on the above regression and then regress the outcome on the treatment and the selected variables you get wrong point estimates and false confidence intervals, like Björn already said.

The purpose for doing this is twofold: comparing your initial model, where variable selection was guided by intuition or theory, to the double-robust selection model gives you an idea as to how good your first model was. Perhaps your first model forgot some important squared or interaction terms and thus suffers from misspecified functional form or omitted variables. Secondly, the Belloni et al. (2014) method can improve inference on your target parameter because redundant regressors were penalized away in their procedure.

| cite | improve this answer | |
  • $\begingroup$ "Correct" point estimates? $\endgroup$ – Richard Hardy Dec 30 '16 at 7:05

To perform a variable selection and then re-run an anslysis, as if no variable selection had happened and the selected model had be intended from the start, typically leads to exaggerated effect sizes, invalid p-values and confidence intervals with below nominal coverage. Perhaps if the sample size is very large and there are a few huge effects and a lot of null effects, LASSO+OLS might not be too badly affected by this, but other than that I cannot see any reasonable justification and in that case the LASSO estimates ought to be just fine, too.

| cite | improve this answer | |
  • 1
    $\begingroup$ But why the second model starts from scratch as if no variable selection happened? Does not LASSO selects explanatory variable with best predictive power? BTW I thought to do stuff LASSO sparse matrix variable into glm again. Now I have understood LASSO per se is a regression. $\endgroup$ – SIslam Apr 1 '17 at 14:01

It may be an excellent idea to run an OLS regression after LASSO. This is simply to double check that your LASSO variable selection made sense. Very often when you rerun the model using OLS regression you uncover that many of the variables selected by LASSO are nowhere near being statistically significant and/or have the wrong sign. And, that may invite you to use another variable selection method that given your data set may be much more robust than LASSO.

LASSO does not always work as intended. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on new data). In reality it very often does the opposite... increase model under-fitting and weakens prediction accuracy. You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO MSE graph." Whenever such graphs show the lowest MSE at the beginning of the X-axis, it shows a LASSO that has failed (increase model under-fitting).

The above unintended consequences are due to the penalty algorithm. Because of it LASSO has no way of distinguishing between a strong causal variable with predictive information and an associated high regression coefficient and a weak variable with no explanatory or predictive information value that has a low regression coefficient. Often, LASSO will prefer the weak variable over the strong causal variable. Also, it may at times even cause to shift the directional signs of variables (shifting from one direction that makes sense to an opposite direction that does not). You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO coefficient path".

| cite | improve this answer | |
  • $\begingroup$ Whoever has downvoted my answer has not digested the abundance of evidence supporting my arguments. Such evidence has mentioned in my answer can be readily uncovered by doing search for Images and doing the following searches: "LASSO MSE graph" and "LASSO coefficient path." After reviewing carefully this body of evidence in conjunction with my arguments, can you please remove your down vote. $\endgroup$ – Sympa Jan 19 at 19:45
  • 1
    $\begingroup$ I suspect the downvote might be in response to the duplication of this post at stats.stackexchange.com/a/445519/919. Please choose which thread is the best fit and delete the superfluous copy. $\endgroup$ – whuber Jan 20 at 14:11
  • 1
    $\begingroup$ That's a good suggestion. And, I'll do that. But, the downvote preceded my duplications. $\endgroup$ – Sympa Jan 20 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.