2
$\begingroup$

If I use the LARS algorithm to fit the LASSO path, is it sufficient to cross-validate using the values of $\lambda$ at each step in LARS or is it better to use a finer grid of $\lambda$ values? I guess I can ask this in two parts:

  • Is the prediction optimal model found at one of the LARS steps?

  • Is the correct subset of variables found at one of the LARS steps?

I would have done the cross-validation over the parameter $s = \left\Vert \beta \right\Vert _{1}\left/ \max \left\Vert \beta \right\Vert _{1}\right.$ on a fine grid with $s\in \left( 0,1\right) $... but then I thought that defeats the purpose of using LARS, which is supposed to be attractive for its lower computational expense. Some clarification would be greatly appreciated.

$\endgroup$
3
$\begingroup$

I have a better understanding of the LASSO and LARS algorithm now so I thought I'd share what I've learned for the benefit of others. I am aware that the pathway between the LARS steps is linear so that, as Donbeo says, linear interpolation can be used to find all LASSO solutions. The point of my question was to ascertain whether this is even necessary. Apparently it is not.

  1. For variable selection - Yes, the best subset occurs at a LARS step. Reasoning:

    • The LAR steps are transition points at which new variables join the active set (as @Donbeo points out), the subset of variables is unchanged between steps. Therefore, if the correct subset is somewhere in the path, then it must occur at a transition point.
    • The LASSO path is shown to contain the true model with high probability (Meinshausen and Buhlmann (2006)).
  2. For prediction - Yes, the optimal $\lambda$ occurs at a LARS step. I've just worked through this paper by Zou, et al (2007), which has answered my question (at least when $X$ has full rank):

    • They show that an unbiased and consistent estimate of the degrees of freedom of the LASSO is the number of variables in the model (or the size of the active set). Hence, all models on the same interval between two LARS steps have the same degrees of freedom.
    • Furthermore, the squared error loss (which we minimize for prediction) is continuous and strictly increasing on each interval. Therefore, the optimal value of $\lambda$ does occur at one of the LARS steps.
$\endgroup$
  • 1
    $\begingroup$ For the 2nd point, the squared error loss in the training set increases, but what about the test loss? i.e., isn't it possible that for a solution at a given step, there is a solution with the same number of variables but more highly penalized, leading to lower test error than the first one? $\endgroup$ – purple51 Oct 27 '14 at 2:21
  • $\begingroup$ Have a look at Theorem 3 on page 11 of Zou, et al (2007). They use AIC or BIC for model selection. These can be seen as estimates of test error, penalizing the training error by a function of the degrees of freedom. For each interval, the training error is increasing and the degrees of freedom is constant, hence the minimum test error should occur at one of the steps. $\endgroup$ – StatGrrl Oct 28 '14 at 8:22
1
$\begingroup$

The LARS algorithm has the property that can detect the exact value of lambda that allows a new variable to join the active set. This does not mean that obtained lambda will be the one that minimize the cross validation.

Two close values of lambda can lead to the same set of active variables but the estimated regression coefficients can be slightly different.

A possible approach that leads to the same solutions of LARS computed on a smaller grid is to fit a linear interpolation between the point obtained with the LARS algorithm

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.