Writing by hand first steps in Least Angle Regression (LARS) How do we write the first steps of Least Angle Regression ? 
What is the rationale behind this method ? What limitations of other methods is it overcoming ? Why is it called Least Angle Regression ?
 A: Least Angle Regression builds a model sequentially, adding a variable at a time. But unlike Forward Stepwise Regression it only adds as much of the predictors as 'it deserves'. Procedure goes as follows.
• Standardize all predictors to have a zero mean and unit variance. Begin with all
regression coefficients at zero i.e. $β_{1} = β_{2} = · · · = β_{p} = 0$. The first residual will be
$r = y − \bar{y}$, since with all $β_{j} = 0$ and standardized predictors the constant coefficient
$β0 = \bar{y}$.
• Set $k = 1$ and begin start the k-th step. Since all values of $β_{j}$ are zero the first residual
is $r_{1} = y − \bar{y}$. Find the predictor xj that is most correlated with this residual r1. Then
as we begin this k = 1 step we have the active step given by $A_{1} = {xj}$ and the active
coefficients given by $β_{A_{1}} = [0]$.
• Move $β_{j}$ from its initial value of 0 and in the direction
$δ_{1} = (X^{T}_{A_{1}}X_{A_{1}})^{−1}X^{T}_{A_{1}} r_{1} = \frac{x^{T}_{j}r_{1}}{x^{
T}
_{j} x_{j}}
= x^
{T}_
{j}
r_{1} .
$
Note that the term $x^{T}_{j} x_{j}$
in the denominator is not present since $x^{
T}_{j} x_{j} = 1$ as all
variables are normalized to have unit variance. The path taken by the elements in $β_{A_{1}}$can be parametrized by $β_{A_{1}}
(α) ≡ β_{A_{1}} + αδ_{1} = 0 + αx^{T}_{j}r_{1} = (x^{
T}_{
j}
r_{1})α $ for $0 ≤ α ≤ 1$ .
• This path of the coefficients $β_{A_{1}}(α)$ will produce a path of fitted values given by $\hat{f_{1}}(α) = X_{A_{1}} β_{A_{1}}(α) = (x^{T}_{
j}
r_{1})α x_{j}$
,
and a residual of
$r(α) = y − \hat{y} − α(x
^{T}_
{j}
r_{1})x_{j} = r_{1} − α(x^{
T}_
{j}
r_{1})x_{j}
$.
Now at this point $x_{j}$
itself has a correlation with this residual as α varies given by
$x^{T}_{
j}
(r_{1} − α(x^{
T}_{
j}
r_{1})x_{j} ) = x^{
T}
_{j}
r_{1} − α(x
^{T}
_{j}
r_{1}) = (1 − α)x
^{T}_{
j}
r_{1} .$
When $α = 0$ this is the maximum value of $x
^{T}_{
j}
r_{1}$ and when $α = 1$ this is the value 0.
All other features (like $x_{k}$) have a correlation with this residual given by
$x^{
T}_{
k}
(r_{1} − α(x^{
T}_
{j}
r_{1})x_{j} ) = x
^{T}
_{k}
r_{1} − α(x
^{T}
_{j}
r_{1})x
^{T}
_{k} x_{j}
$.
The fit vector at step k evolves as $\hat{f_{k}}(\alpha)=\hat{f_{k}}+\alpha u_{k}$ where $u_{k}=X_{A_{k}}\delta_{k}$ is the new fit direction. We can show that $u_{k}$ makes the smallest angle with each of the predictors in $A_{k}$. Hence the name of the procedure.  
Stepwise forward selection might be an overly greedy algorithm and as such might discard a predictor which is in fact strongly correlated with $x_{1}$. Forward stagewise selection is a much more cautious procedure taking tiny steps towards solution. LAR is an alternative with larger steps and reduced computational burden. LAR is tighly coupled with Lasso - LAR and Lasso will give same coefficients estimation unless a non-zero coefficient reach zero. A variant of LAR exist to mimic and compute the Lasso estimation. 
