How does it make sense to do OLS after LASSO variable selection?

Question

Recently I have found that in the applied econometrics literature, when dealing with feature selection problems, it is not uncommon to perform LASSO followed by an OLS regression using the selected variables.

I was wondering how can we qualify the validness of such a procedure. Will it cause troubles such as omitted variables? Any proofs showing that it is more efficient, or the results are more interpretable?

Here are some related discussions:

Variable selection with LASSO

Using trees after variable selection using Lasso/Random

If, as pointed out, such a procedure is not correct in general, then why are still there so many researches doing so? Can I say that it is just a rule of thumb, a compromise solution, due to some of the uneasy properties of LASSO estimator, and people's fondness towards OLS?

Answer 1

There was a similar question a few days ago which had the relevant reference:

• Belloni, A., Chernozhukov, V., and Hansen, C. (2014) "Inference on Treatment Effects after Selection among High-Dimensional Controls", Review of Economic Studies, 81(2), pp. 608-50 (link)

At least for me the paper is a pretty tough read because the proofs behind this relatively simple are fairly elaborate. When you are interested in estimating a model like $$y_i = \alpha T_i + X_i'\beta + \epsilon_i$$

where $y_i$ is your outcome, $T_i$ is some treatment effect of interest, and $X_i$ is a vector of potential controls. The target parameter is $\alpha$. Assuming that most of the variation in your outcome is explained by the treatment and a sparse set of controls, Belloni et al. (2014) develop a double-robust selection method which provides correct point estimates and valid confidence intervals. This sparsity assumption is important though.

If $X_i$ includes a few important predictors of $y_i$ but you don't know which they are (either single variables, their higher order polynomials, or interactions with other variables), you can perform a three step selection procedure:

1. regress $y_i$ on $X_i$, their squares, and interactions, and select important predictors using LASSO
2. regress $T_i$ on $X_i$, their squares, and interactions, and select important predictors using LASSO
3. regress $y_i$ on $T_i$ and all the variables which were selected in either of the first two steps

They provide proofs as for why this works and why you get the correct confidence intervals etc out of this method. They also show that if you only perform a LASSO selection on the above regression and then regress the outcome on the treatment and the selected variables you get wrong point estimates and false confidence intervals, like Björn already said.

The purpose for doing this is twofold: comparing your initial model, where variable selection was guided by intuition or theory, to the double-robust selection model gives you an idea as to how good your first model was. Perhaps your first model forgot some important squared or interaction terms and thus suffers from misspecified functional form or omitted variables. Secondly, the Belloni et al. (2014) method can improve inference on your target parameter because redundant regressors were penalized away in their procedure.

Answer 2

To perform a variable selection and then re-run an anslysis, as if no variable selection had happened and the selected model had be intended from the start, typically leads to exaggerated effect sizes, invalid p-values and confidence intervals with below nominal coverage. Perhaps if the sample size is very large and there are a few huge effects and a lot of null effects, LASSO+OLS might not be too badly affected by this, but other than that I cannot see any reasonable justification and in that case the LASSO estimates ought to be just fine, too.

Answer 3

It may be an excellent idea to run an OLS regression after LASSO. This is simply to double check that your LASSO variable selection made sense. Very often when you rerun the model using OLS regression you uncover that many of the variables selected by LASSO are nowhere near being statistically significant and/or have the wrong sign. And, that may invite you to use another variable selection method that given your data set may be much more robust than LASSO.

LASSO does not always work as intended. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on new data). In reality it very often does the opposite... increase model under-fitting and weakens prediction accuracy. You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO MSE graph." Whenever such graphs show the lowest MSE at the beginning of the X-axis, it shows a LASSO that has failed (increase model under-fitting).

The above unintended consequences are due to the penalty algorithm. Because of it LASSO has no way of distinguishing between a strong causal variable with predictive information and an associated high regression coefficient and a weak variable with no explanatory or predictive information value that has a low regression coefficient. Often, LASSO will prefer the weak variable over the strong causal variable. Also, it may at times even cause to shift the directional signs of variables (shifting from one direction that makes sense to an opposite direction that does not). You can see many examples of that by searching the Internet for Images and searching specifically for "LASSO coefficient path".