Skip to main content
4 of 6
added 6 characters in body
yliueagle
  • 855
  • 2
  • 8
  • 10

Why use Lasso estimates over OLS estimates on variable subset?

For Lasso regression $L(\beta)=(X\beta-y)'(X\beta-y)+\lambda||\beta||_1$, suppose the best solution (minimum testing error for example) selects $k$ features, so that $\hat{\beta}^{lasso}=\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso},0,...0\right)$. We know $\left(\hat{\beta}_1^{lasso},\hat{\beta}_2^{lasso},...,\hat{\beta}_k^{lasso}\right)$ is a biased estimate of $\left(\beta_1,\beta_2,...,\beta_k\right)$, why we still take $\hat{\beta}^{lasso}$ as the final solution, instead of the more 'reasonable' $\hat{\beta}^{new}=\left(\hat{\beta}_{1:k}^{new},0,...,0\right)$, where $\hat{\beta}_{1:k}^{new}$ is the LS estimate from partial model $L^{new}(\beta_{1:k})=(X_{1:k}*\beta-y)'(X_{1:k}*\beta-y)$. ($X_{1:k}$: keep the columns of $X$ corresponding to the $k$ selected features). In brief, why we use Lasso for both feature election and parameter estimation, instead of only for variable selection and leaving the estimation by other models on the selected features?

More over, what does 'Lasso can select at most $n$ features'? $n$ is the sample size.

yliueagle
  • 855
  • 2
  • 8
  • 10