For Lasso regression $$L(\beta)=(X\beta-y)'(X\beta-y)+\lambda\|\beta\|_1,$$ suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso},0,...0\right)$.

We know that $\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso}\right)$ is a biased estimate of $\left(\beta_1,\beta_2,...,\beta_k\right)$, so why do we still take $\hat{\beta}^{lasso}$ as the final solution, instead of the more 'reasonable' $\hat{\beta}^{new}=\left(\hat{\beta}_{1:k}^{new},0,...,0\right)$, where $\hat{\beta}_{1:k}^{new}$ is the LS estimate from partial model $L^{new}(\beta_{1:k})=(X_{1:k}\beta-y)'(X_{1:k}\beta-y)$. ($X_{1:k}$ denotes the columns of $X$ corresponding to the $k$ selected features).

In brief, why do we use Lasso both for feature selection and for parameter estimation, instead of only for variable selection (and leaving the estimation on the selected features to OLS)?

(Also, what does it mean that 'Lasso can select at most $n$ features'? $n$ is the sample size.)

  • 1
    $\begingroup$ That is a very good question. Have you tried a few simulations to see how different the results would be from standard Lasso if you one tried it your way? $\endgroup$
    – Placidia
    Commented Jan 16, 2014 at 14:38
  • 3
    $\begingroup$ Did you understand the purpose of "Shrinkage" in LASSO? $\endgroup$
    – Michael M
    Commented Jan 16, 2014 at 15:02
  • 7
    $\begingroup$ The idea's to shrink the coefficient estimates precisely because you've picked the biggest ones. Least-squares estimates are no longer unbiased when you've done feature selection beforehand. $\endgroup$ Commented Jan 16, 2014 at 15:07
  • 2
    $\begingroup$ See the following question for a great answer to "What problem do shrinkage methods solve?" stats.stackexchange.com/questions/20295/… $\endgroup$
    – D L Dahly
    Commented Jan 16, 2014 at 16:26
  • 2
    $\begingroup$ To be clear: Not saying @Scortchi is wrong, but this is a bit of a grey area when discussing feature selection, and I think this is an important technical point that should be made very clear. $\endgroup$
    – JohnA
    Commented May 13, 2015 at 18:20

3 Answers 3


I don't believe there is anything wrong with using LASSO for variable selection and then using OLS. From "Elements of Statistical Learning" (pg. 91)

...the lasso shrinkage causes the estimates of the non-zero coefficients to be biased towards zero and in general they are not consistent [Added Note: This means that, as the sample size grows, the coefficient estimates do not converge]. One approach for reducing this bias is to run the lasso to identify the set of non-zero coefficients, and then fit an un-restricted linear model to the selected set of features. This is not always feasible, if the selected set is large. Alternatively, one can use the lasso to select the set of non-zero predictors, and then apply the lasso again, but using only the selected predictors from the first step. This is known as the relaxed lasso (Meinshausen, 2007). The idea is to use cross-validation to estimate the initial penalty parameter for the lasso, and then again for a second penalty parameter applied to the selected set of predictors. Since the variables in the second step have less "competition" from noise variables, cross-validation will tend to pick a smaller value for $\lambda$ [the penalty parameter], and hence their coefficients will be shrunken less than those in the initial estimate.

Another reasonable approach similar in spirit to the relaxed lasso, would be to use lasso once (or several times in tandem) to identify a group of candidate predictor variables. Then use best subsets regression to select the best predictor variables to consider (also see "Elements of Statistical Learning" for this). For this to work, you would need to refine the group of candidate predictors down to around 35, which won't always be feasible. You can use cross-validation or AIC as a criterion to prevent over-fitting.

  • $\begingroup$ Another part of my question is, why 'Lasso can select at most n features'? If this is the case, I think OLS on the selected features will be at least 'good', since OLS is the 'BLUE' (Not strictly BLUE since it's mostly biased). Just consider an extreme situation that Lasso selects the exactly right features, conducting OLS on these features will restore the true model, which I think is better than the Lasso estimation. $\endgroup$
    – yliueagle
    Commented Jan 23, 2014 at 15:22
  • 2
    $\begingroup$ The problem is that this "extreme situation" is very unlikely to occur, and there is no way of knowing if LASSO has selected exactly the right features. If LASSO selects too many features, then I think the full OLS model may perform worse than the LASSO estimates. Similarly, ridge regression can outperform OLS if there are too many features (i.e. OLS is overfit). $\endgroup$ Commented Jan 23, 2014 at 18:25
  • 2
    $\begingroup$ See also web.stanford.edu/~hastie/StatLearnSparsity_files/SLS.pdf, the end of Section 2.2: "[...] the least-squares fit on the subset of [...] predictors tends to expand the lasso estimates away from zero. The nonzero estimates from the lasso tend to be biased toward zero, so the debiasing in the right panel can often improve the prediction error of the model. This two-stage process is also known as the relaxed lasso (Meinshausen 2007)." $\endgroup$
    – amoeba
    Commented May 30, 2017 at 12:20
  • 1
    $\begingroup$ I looked into Meinshausen paper and it actually recommends fitting two penalty parameters, as described in your original quote from The Elements. +1 $\endgroup$
    – amoeba
    Commented May 30, 2017 at 12:29
  • $\begingroup$ @AlexWilliams But isn't there a sparsity assumption in the previous paragraph about the correlation between the selected set and what is removed being small? $\endgroup$
    – dimitriy
    Commented Oct 18, 2017 at 17:47

If your aim is optimal in-sample performance (wrt highest R-squared), then just use OLS on every available variable. Dropping variables will decrease R-squared.

If your aim is good out-of-sample performance (which is usually what is much more important), then your proposed strategy will suffer from two sources of overfitting:

  • Selection of variables based on correlations with the response variable
  • OLS estimates

The purpose of LASSO is to shrink parameter estimates towards zero in order to fight above two sources of overfitting. In-sample predictions will be always worse than OLS, but the hope is (depending on the strength of the penalization) to get more realistic out-of-sample behaviour.

Regarding $p > n$: This (probably) depends on the implementation of LASSO you are using. A variant, Lars (least angle regression), does easily work for $p > n$.

  • 2
    $\begingroup$ The "Leekasso" (always pick 10 coefficients) is different than the question's proposal (re-estimate OLS with k predictors picked by LASSO) $\endgroup$
    – Affine
    Commented Jan 16, 2014 at 15:42
  • $\begingroup$ @affine you are completely right. I removed the reference. $\endgroup$
    – Michael M
    Commented Jan 16, 2014 at 15:48
  • 3
    $\begingroup$ This sounds reasonable, but the inventors of Lasso argue otherwise and actually recommend using two-stage procedure with OLS on the Lasso-identified subset (as suggested by the OP), see @Alex'es answer. $\endgroup$
    – amoeba
    Commented May 30, 2017 at 12:31
  • $\begingroup$ I like this answer because it mentions the selection bias from the search itself; it sure feels like there should be an additional penalty. LASSO as mere subset selection mechanism - is that all it is? Then why even print out its coefficients at all? $\endgroup$
    – Ben Ogorek
    Commented Nov 14, 2019 at 15:33

Regarding the OPs question of why Lasso can select at most n features:

Consider why an OLS might be biased: this is when there are more predictors (p) than observations (n). Thus $X^{T}X$ is of size [p,p] in $\beta = (X^{T} X)^{-1}X^{T}Y$. Taking an inverse of such a matrix is not possible (it may be singular).

Lasso is forced to shrink the coefficients of the variables so that this does not happen, thus it never selects more than n features so that $X^{T}X$ is always invertible.

  • 1
    $\begingroup$ (-1) I don't think this is true. Can you explain more the connection between $(X^TX)^{-1}$ not existing and the lasso? Specifically, what does $X^TX have to do with the lasso? There are proofs of the OPS question (the answers here are revealing, for instance: stats.stackexchange.com/questions/38299/…) but this answer doesn't appear prove it. (Please let me know if I'm mistaken!) $\endgroup$
    – user795305
    Commented May 30, 2017 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.