2
$\begingroup$

When do Lasso and OLS give the same solution when applied independently from each other?

But more importantly for me: If you first run Lasso and say X* are the features that Lasso has not shrunk to zero (so the features that Lasso has kept in the model with a non-zero coefficient in front of it). And then run OLS with these X* (excluding the features that Lasso has shrunk) will you then get the same results (e.g. same coefficients) as the Lasso?

I have this intuition that you will not get the same, but I cannot fully understand why. I assume the L1 term also has effected the estimation of the coefficients of X*? Loosely speaking that observations were used to shrink and thus it is not same as using all observations to estimate the coefficients of X* in an OLS setting?

$\endgroup$

2 Answers 2

1
$\begingroup$

tl;dr: LASSO is incredibly versatile, but will have different results from OLS. There are many ways to apply LASSO that are used in practice including:

  1. LASSO off the bat
  2. LASSO for feature selection then OLS
  3. LASSO for feature selection then picking a different value of $\lambda$ for regression

First, LASSO and OLS will not give the same result unless $\lambda = 0$. They will be minimizing different loss functions (as pointed out by user2974951) and the penalty $\lambda \|\beta\|_1$ will always be pulling it away from the OLS estimate (this can be proved by the fact that you can get the LASSO estimates can result as a Bayes estimate with proper choice of prior and any Bayes estimate with proper prior is biased. OLS, however, is unbiased).

Your intuition is correct. This process of first using LASSO to select features and then running normal OLS on the new model is actually done frequently (when I talked to Trevor Hastie about it he called it "hard LASSOing"), but it is different from just running LASSO and taking the coefficients.

Generally, we can think of LASSO as translating coefficients by a factor $\lambda$ but truncating at zero (this is exactly what happens when $X$ is orthogonal and can be useful to help think about how to think of the effect of LASSO). If we were translating all of the coefficients before, get rid of some of them, and then run a regression without the penalty but on the new variables, then we will get different coefficients from the LASSO estimates because the LASSO estimates included the translation towards zero while the new OLS estimates will not.

Edit: The idea behind using LASSO for just the variable selection step, but not the regression overall is to get consistent estimators (that is, we want to say that as sample size goes to infinity, the estimates converge to the true values, this is generally not true of LASSO).

Another common trick is to do two-stage LASSO or "relaxed LASSO" where you first pick $\lambda$ via cross-validation to do variable selection and then do LASSO again via cross-validation, but after having trimmed the number of features you have. This will lead to lower values of $\lambda$ than with just straight LASSO (so the estimates will likely be closer to the true values), but will also have better generalizability to new data.

$\endgroup$
3
  • 2
    $\begingroup$ While this answers the narrow question whether coefficients will be different, it does not address the deeper issue whether this is a reasonable approach. One advantage of LASSO regression is that it better generalizes to unseen data than other variable selection methods (like optimizing AIC or cross-validated $R^2$). I wonder whether this nice property gets lost when LASSO is only used for variable selection but its coefficients are discarded. $\endgroup$
    – cdalitz
    Commented May 24, 2022 at 9:27
  • 1
    $\begingroup$ Generally yes, you would lose some of the regularization benefits of LASSO if you do OLS after variable selection, but it would generally still lead to more parsimonious/smaller models than AIC and adjusted $R^2$ simply because reasonable values of $\lambda$ from cross validation would punish adding parameters more. $\endgroup$
    – PhysicsKid
    Commented May 24, 2022 at 9:39
  • $\begingroup$ @cdalitz, comparing LASSO to optimizing AIC or cross-validated $R^2$ is an apples to oranges comparison. LASSO is an estimation and variable selection technique while AIC and $R^2$ are model evaluation metrics. LASSO is sometimes used in combination with either of the two. $\endgroup$ Commented May 24, 2022 at 10:02
0
$\begingroup$

The formula for LASSO is

$$\frac{1}{N}||y-X\beta||_2^2+\lambda||\beta||_1$$

Aside from the constant $1/N$, the only difference is the second term $\lambda||\beta||_1$. It is this parameter $\lambda$ which you optimize during lasso and which determines shrinkage. Normally you would choose this using cross validation. Setting $\lambda=0$ (and hence the whole term being equal to 0) you would get the same result as OLS.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.