# Writing by hand first steps in Least Angle Regression (LARS)

How do we write the first steps of Least Angle Regression ?

What is the rationale behind this method ? What limitations of other methods is it overcoming ? Why is it called Least Angle Regression ?

Least Angle Regression builds a model sequentially, adding a variable at a time. But unlike Forward Stepwise Regression it only adds as much of the predictors as 'it deserves'. Procedure goes as follows.

• Standardize all predictors to have a zero mean and unit variance. Begin with all regression coefficients at zero i.e. $$β_{1} = β_{2} = · · · = β_{p} = 0$$. The first residual will be $$r = y − \bar{y}$$, since with all $$β_{j} = 0$$ and standardized predictors the constant coefficient $$β0 = \bar{y}$$.

• Set $$k = 1$$ and begin start the k-th step. Since all values of $$β_{j}$$ are zero the first residual is $$r_{1} = y − \bar{y}$$. Find the predictor xj that is most correlated with this residual r1. Then as we begin this k = 1 step we have the active step given by $$A_{1} = {xj}$$ and the active coefficients given by $$β_{A_{1}} = [0]$$.

• Move $$β_{j}$$ from its initial value of 0 and in the direction $$δ_{1} = (X^{T}_{A_{1}}X_{A_{1}})^{−1}X^{T}_{A_{1}} r_{1} = \frac{x^{T}_{j}r_{1}}{x^{ T} _{j} x_{j}} = x^ {T}_ {j} r_{1} .$$ Note that the term $$x^{T}_{j} x_{j}$$ in the denominator is not present since $$x^{ T}_{j} x_{j} = 1$$ as all variables are normalized to have unit variance. The path taken by the elements in $$β_{A_{1}}$$can be parametrized by $$β_{A_{1}} (α) ≡ β_{A_{1}} + αδ_{1} = 0 + αx^{T}_{j}r_{1} = (x^{ T}_{ j} r_{1})α$$ for $$0 ≤ α ≤ 1$$ .

• This path of the coefficients $$β_{A_{1}}(α)$$ will produce a path of fitted values given by $$\hat{f_{1}}(α) = X_{A_{1}} β_{A_{1}}(α) = (x^{T}_{ j} r_{1})α x_{j}$$ , and a residual of $$r(α) = y − \hat{y} − α(x ^{T}_ {j} r_{1})x_{j} = r_{1} − α(x^{ T}_ {j} r_{1})x_{j}$$.

Now at this point $$x_{j}$$ itself has a correlation with this residual as α varies given by $$x^{T}_{ j} (r_{1} − α(x^{ T}_{ j} r_{1})x_{j} ) = x^{ T} _{j} r_{1} − α(x ^{T} _{j} r_{1}) = (1 − α)x ^{T}_{ j} r_{1} .$$ When $$α = 0$$ this is the maximum value of $$x ^{T}_{ j} r_{1}$$ and when $$α = 1$$ this is the value 0. All other features (like $$x_{k}$$) have a correlation with this residual given by $$x^{ T}_{ k} (r_{1} − α(x^{ T}_ {j} r_{1})x_{j} ) = x ^{T} _{k} r_{1} − α(x ^{T} _{j} r_{1})x ^{T} _{k} x_{j}$$.

The fit vector at step k evolves as $$\hat{f_{k}}(\alpha)=\hat{f_{k}}+\alpha u_{k}$$ where $$u_{k}=X_{A_{k}}\delta_{k}$$ is the new fit direction. We can show that $$u_{k}$$ makes the smallest angle with each of the predictors in $$A_{k}$$. Hence the name of the procedure.

Stepwise forward selection might be an overly greedy algorithm and as such might discard a predictor which is in fact strongly correlated with $$x_{1}$$. Forward stagewise selection is a much more cautious procedure taking tiny steps towards solution. LAR is an alternative with larger steps and reduced computational burden. LAR is tighly coupled with Lasso - LAR and Lasso will give same coefficients estimation unless a non-zero coefficient reach zero. A variant of LAR exist to mimic and compute the Lasso estimation.