LASSO and related path algorithms I want to understand why LASSO is called a path algorithm. There are also so many related path algorithms that have sprung out: 
incremental forward stagewise regression, piecewise-linear path algorithms, dantzig selector, grouped lasso.
All of these are found in the book elements of statistical learning. In my opinion, the way the book is written is not very friendly to those trying to understand these groundbreaking concepts for the first time. 
Can anyone kindly explain those algorithms written above? And perhaps also give a link to more elementary discussions? 
Thanks
 A: The Lasso is the solution to 
$$\hat{\beta} \in argmin_\beta \frac{1}{2n}\Vert y-X\beta\Vert_2^2 + \lambda\Vert\beta\Vert_1$$
Evidently, $\hat{\beta}$ also depends on $\lambda$, so really, we could write this dependence explicitly: $\hat{\beta}(\lambda)$. Thus, we can interpret $\hat{\beta}$ as a function $\lambda\mapsto\hat{\beta}(\lambda)$, whose domain is $[0,\infty)$. The "solution path" of the Lasso is precisely this function: For each $\lambda\in[0,\infty)$, there is a vector $\hat{\beta}(\lambda)$ that solves the optimization problem above. When $\lambda=0$, you start out at the OLS solution, and as you increase $\lambda$, the Lasso solution changes to become more and more sparse, until ultimately $\lambda=+\infty$ and $\hat{\beta}(+\infty) = 0$.
Many approximate algorithms for the Lasso solve this optimization problem at some discrete values $\lambda_1,\ldots,\lambda_L$. This is simple and efficient, but it does not provide the full spectrum of possible solutions, since there are infinitely many possible choices of $\lambda$.
LARS, and other related algorithms, on the other hand, provide the full solution path for all possible $\lambda$ (i.e. infinitely many solutions). This why they are commonly referred to as "path algorithms" since they provide a parametrized solution "path" for all $\lambda$. 
Note: To understand the use of the word "path", think of parametric equations and lines from calculus e.g. $f(t) = (x(t), y(t))$. As $t$ varies, $f(t)$ plots a path in space. For the Lasso, replace $t$ with $\lambda$ and we have $\hat{\beta}(\lambda) = (\hat{\beta}_1(\lambda),\ldots,\hat{\beta}_p(\lambda))$.
