Stopping condition for least-angle regression Suppose I have data with p explanatory variables, and I want to use a LARS algorithm to build a model. Do I


*

*Run LARS until all p variables have been added to my model, and the correlation between each variable and the residual is 0?

*Or do I terminate LARS earlier, say a) by stopping LARS once k < p variables have been added to my model, or b) by stopping when the correlation between each variable and the residual crosses below some threshold t? (Where k or t could be determined by cross-validation.)


In reading the original paper by Efron et al. again, it sounds like I do version 1 (build the full model). But then I'm confused -- how does this solution differ from the full maximum-likelihood OLS estimate? (The OLS estimate also produces zero correlation between each variable and the residual, and I thought there was only a single way to get this zero correlation, i.e., by orthogonal projection. Or am I mistaken?)
 A: Certainly, if $p \leq n$ and you run LARS until you've included all $p$ variables in the model and the correlations are zero, then the solution will be exactly the OLS solution.
You can view LARS as just another "regularized" least-squares estimate. Of course, it has a very close connection to both forward-stagewise regression and the lasso. My suspicion is that most people use the LARS algorithm primarily to compute the lasso solutions and that LARS itself has gotten fairly little use as an estimation method in its own right.
Cross-validation to choose $k$ and $t$ should be feasible if your interest is on prediction. Since the maximal correlation decreases monotonically, then $k$ is completely determined by $t$ in LARS (see, e.g., the right pane of Fig. 3 of the paper). Hence, you only need optimize over the threshold $t$ in the cross-validation. This is cleaner in a couple of ways: (1) There is only one parameter to optimize over and (2) any sensible objective function you choose should be pretty much continuous in $t$, whereas $k$ is discrete. Discrete-valued parameters can often be less pleasant to deal with in CV.
For more details regarding some of the properties of least angle regression, you can also see the solution to this question.
