If we select different values of the parameter $\lambda$, we could obtain solutions with different sparsity levels. Does it mean the regularization path is how to select the coordinate that could get faster convergence? I'm a little confused although I have heard about sparsity often. In addition, could you please give a simple description about the existing solutions of LASSO problem?


Say you have a model with $p$ predictor variables: $x_1, x_2, \ldots x_p$. Set $\lambda$ to an initial value, and estimate your coefficients $\beta_1, \beta_2, \ldots \beta_p$. These coefficients can be thought of as a point in $p$-dimensional space.*

Repeat the procedure for your next value of $\lambda$, and get another set of estimates. These form another point in $p$-dimensional space. Do this for all your $\lambda$ values, and you will get a sequence of such points. This sequence is the regularization path.

* There's also the intercept term $\beta_0$ so all this technically takes place in $(p+1)$-dimensional space, but never mind that. Anyway most elastic net/lasso programs will standardise the variables before fitting the model, so $\beta_0$ will always be 0.


A graphical explanation of the Lasso solution can be found on pages 69-73 of the text "Elements of Statistical Learning" (online version here).

  • 1
    $\begingroup$ Thank you,sir! Yes! The Fig. 3.10 is the lasso regularization path. $\endgroup$ – mining Aug 21 '14 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.