# Why use Lasso estimates over OLS estimates on variable subset?

For Lasso regression $L(\beta)=(X\beta-y)'(X\beta-y)+\lambda*norm(\beta,1)$, suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso},0,...0\right)$. We know $\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso}\right)$ is biased estimate of $\left(\beta_1,\beta_2,...,\beta_k\right)$, why we still take $\hat{\beta}^{lasso}$ as our final solution, but not the more 'reasonable' one $\hat{\beta}^{new}=\left(\hat{\beta}_{1:k}^{new},0,...,0\right)$, where $\hat{\beta}_{1:k}^{new}$ is the LS estimate from partial model $L^{new}(\beta_{1:k})=(X_{1:k}*\beta-y)'(X_{1:k}*\beta-y)$. ($X_{1:k}$: keep the columns of $X$ corresponding to the $k$ selected features). In brief, why we use Lasso for both feature election and estimation, instead of only for variable selection and leaving the estimation by other models on the selected features?

More over, what does 'Lasso can select at most $n$ features'? $n$ is the sample size.

-
That is a very good question. Have you tried a few simulations to see how different the results would be from standard Lasso if you one tried it your way? –  Placidia Jan 16 at 14:38
Did you understand the purpose of "Shrinkage" in LASSO? –  Michael Mayer Jan 16 at 15:02
@MichaelMayer The purpose of 'Shrinkage' in Lasso is to decrease feature size for both better interpretation and prediction, I think –  yliueagle Jan 16 at 15:06
The idea's to shrink the coefficient estimates precisely because you've picked the biggest ones. Least-squares estimates are no longer unbiased when you've done feature selection beforehand. –  Scortchi Jan 16 at 15:07
You should look at the Adaptive Lasso. I guess you will be pleased to see that one. Uses weights based on ols estimates to reduce bias. –  Scratch Jan 16 at 16:16

If your aim is optimal in-sample performance (wrt highest R-squared), then just use OLS on every available variable. Dropping variables will decrease R-squared.

If your aim is good out-of-sample performance (which is usually what is much more important), then your proposed strategy will suffer from two sources of overfitting:

• Selection of variables based on correlations with the response variable
• OLS estimates

The purpose of LASSO is to shrink parameter estimates towards zero in order to fight above two sources of overfitting. In-sample predictions will be always worse than OLS, but the hope is (depending on the strength of the penalization) to get more realistic out-of-sample behaviour.

Regarding $p > n$: This (probably) depends on the implementation of LASSO you are using. A variant, Lars (least angle regression), does easily work for $p > n$.

-
The "Leekasso" (always pick 10 coefficients) is different than the question's proposal (re-estimate OLS with k predictors picked by LASSO) –  Affine Jan 16 at 15:42
@affine you are completely right. I removed the reference. –  Michael Mayer Jan 16 at 15:48
...the lasso shrinkage causes the estimates of the non-zero coefficients to be biased towards zero and in general they are not consistent [Added Note: This means that, as the sample size grows, the coefficient estimates do not converge]. One approach for reducing this bias is to run the lasso to identify the set of non-zero coefficients, and then fit an un-restricted linear model to the selected set of features. This is not always feasible, if the selected set is large. Alternatively, one can use the lasso to select the set of non-zero predictors, and then apply the lasso again, but using only the selected predictors from the first step. This is known as the relaxed lasso (Meinshausen, 2007). The idea is to use cross-validation to estimate the initial penalty parameter for the lasso, and then again for a second penalty parameter applied to the selected set of predictors. Since the variables in the second step have less "competition" from noise variables, cross-validation will tend to pick a smaller value for $\lambda$ [the penalty parameter], and hence their coefficients will be shrunken less than those in the initial estimate.