5
$\begingroup$

I want to understand why LASSO is called a path algorithm. There are also so many related path algorithms that have sprung out:

incremental forward stagewise regression, piecewise-linear path algorithms, dantzig selector, grouped lasso.

All of these are found in the book elements of statistical learning. In my opinion, the way the book is written is not very friendly to those trying to understand these groundbreaking concepts for the first time.

Can anyone kindly explain those algorithms written above? And perhaps also give a link to more elementary discussions?

Thanks

$\endgroup$
8
  • $\begingroup$ Does this help? Tibshirani and Taylor, The solution path of the generalized lasso $\endgroup$ Commented May 18, 2015 at 9:04
  • $\begingroup$ Thanks for your help, but I am looking for at least a layman's explanation to all these methods. These are overwhelming to first timers like me. The problem is, I do not understand what is happening (or what problem they are trying to solve). The textbook is not too intuitive either. $\endgroup$
    – cgo
    Commented May 18, 2015 at 9:07
  • $\begingroup$ it was mostly the title and abstract that I wanted you to see. You're right that Elements is better for reference than learning. You might be better off with An introduction to statistical learning by James, Witten, Hastie, and Tibshirani $\endgroup$ Commented May 18, 2015 at 9:11
  • $\begingroup$ Yes, I have that textbook. The writing is very intuitive and it is an excellent resource. However, the algorithms listed above aren't listed in the Introduction to Stat Learning Book. I do not know where to find more tutorial/introductory resource. $\endgroup$
    – cgo
    Commented May 18, 2015 at 9:13
  • $\begingroup$ I'm not sure myself. I will say that these papers are intimidating up pick up, but the math isn't always so bad. A lot of the time it turns out to just be very clever applications of college level calculus and linear algebra $\endgroup$ Commented May 18, 2015 at 9:18

1 Answer 1

7
$\begingroup$

The Lasso is the solution to

$$\hat{\beta} \in argmin_\beta \frac{1}{2n}\Vert y-X\beta\Vert_2^2 + \lambda\Vert\beta\Vert_1$$

Evidently, $\hat{\beta}$ also depends on $\lambda$, so really, we could write this dependence explicitly: $\hat{\beta}(\lambda)$. Thus, we can interpret $\hat{\beta}$ as a function $\lambda\mapsto\hat{\beta}(\lambda)$, whose domain is $[0,\infty)$. The "solution path" of the Lasso is precisely this function: For each $\lambda\in[0,\infty)$, there is a vector $\hat{\beta}(\lambda)$ that solves the optimization problem above. When $\lambda=0$, you start out at the OLS solution, and as you increase $\lambda$, the Lasso solution changes to become more and more sparse, until ultimately $\lambda=+\infty$ and $\hat{\beta}(+\infty) = 0$.

Many approximate algorithms for the Lasso solve this optimization problem at some discrete values $\lambda_1,\ldots,\lambda_L$. This is simple and efficient, but it does not provide the full spectrum of possible solutions, since there are infinitely many possible choices of $\lambda$.

LARS, and other related algorithms, on the other hand, provide the full solution path for all possible $\lambda$ (i.e. infinitely many solutions). This why they are commonly referred to as "path algorithms" since they provide a parametrized solution "path" for all $\lambda$.

Note: To understand the use of the word "path", think of parametric equations and lines from calculus e.g. $f(t) = (x(t), y(t))$. As $t$ varies, $f(t)$ plots a path in space. For the Lasso, replace $t$ with $\lambda$ and we have $\hat{\beta}(\lambda) = (\hat{\beta}_1(\lambda),\ldots,\hat{\beta}_p(\lambda))$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.