5
$\begingroup$

I need some help to understand the relationship between the ranking of the variables from the LARS algorithm and the use of OLS to estimate the final model chosen by the LARS.

I understand that the LARS algorithm is less greedy than forward stepwise regression because it does not require additional predictors to be orthogonal to the residual and the already included predictor. But after the LARS has ranked the variables and chosen the optimal number of predictors to include in the model, we use OLS to estimate the model. The OLS parameters are different from those assigned to the predictors in the LARS, right? So how can I intuitively explain why it is correct to first use LARS and then OLS on the model selected by LARS?

Improve this question
$\endgroup$
2
  • 6
    $\begingroup$ I don't think this is often done. Where did you find this? It is mentioned in one of the papers from mid-2000s as a possible solution, but I do not think it is standard practice. $\endgroup$
    – Richard Hardy
    Commented Jun 15, 2017 at 11:34
  • 2
    $\begingroup$ It is described in the section "LARS–OLS hybrid" in Efron et. al (2004). In addition, this is the procedure in Gelper and Croux (2008). They use LARS for time series, which is what I do as well. $\endgroup$
    – Guest
    Commented Jun 15, 2017 at 11:56

1 Answer 1

Reset to default
9
$\begingroup$

The coefficient estimates from LARS will be shrunk (biased) towards zero, and the intensity of shrinkage might be suboptimal (too harsh) for forecasting.

However, some shrinkage should be good, as there is a trade-off between bias and variance. For example, if lasso happens to have selected the relevant regressors and only them (which of course is never guaranteed in practice), you could show that a positive (thus nonzero) amount of ridge-type shrinkage is optimal* -- just as you can show it in a basic linear model with no variable selection (see e.g. the answer by Andrew M in the thread "Under exactly what conditions is ridge regression able to provide an improvement over ordinary least squares regression?"). (I do not know if you can show this for LARS-type shrinkage, but intuitively I would not expect zero shrinkage to always be optimal.)

This is what motivates (1) relaxed lasso (Meinshausen, 2007) where there are two shrinkage parameters: a harsher one for variable selection and a softer one of the coefficients of the retained variables); or (2) LARS-OLS where there is no shrinkage on the coefficients of the retained variables.

*Optimal in the sense that it minimizes the mean squared error of the estimator

Meinshausen, Nicolai. "Relaxed lasso." Computational Statistics & Data Analysis 52.1 (2007): 374-393.

Improve this answer
$\endgroup$
16
  • $\begingroup$ could you clarify what you mean by "optimal"? In particular, are you referencing a theorem you could point me towards? $\endgroup$
    – user795305
    Commented Jun 15, 2017 at 14:03
  • 1
    $\begingroup$ @Ben, thanks for your remark. I have now explained the sense in which the estimator is optimal. I have also included a link to Dave Giles' blog post for a broader introduction to the topic. $\endgroup$
    – Richard Hardy
    Commented Jun 15, 2017 at 14:15
  • $\begingroup$ While I'm hoping this isn't too nitgritty of an objection: it doesn't appear that what you link to supports what you're saying. From my understanding, the blog post is motivation for james stein estimation in terms of regression. On the other hand, you're talking about ridge regression. These are different types of shrinkage. To go further, ridge regression depends on a tuning param that needs to be selected in a data driven way. In the presence of tuning, I'm not aware of any theory whatsoever regarding the risk of the resulting estimator. – Ben 7 mins ago $\endgroup$
    – user795305
    Commented Jun 15, 2017 at 15:05
  • $\begingroup$ Indeed, Hastie + Efron recently wrot ein ch 7.3 of Computer age statistical inference that "The main point here is that at present there is no optimality theory for shrinkage estimation. Fisher provided an elegant theory for optimal unbiased estimation. It remains to be seen whether biased estimation can be neatly codified." Please let me know if I'm not thinking about this right! $\endgroup$
    – user795305
    Commented Jun 15, 2017 at 15:06
  • 1
    $\begingroup$ @Ben, thanks for your input. You probably know more than me about these things. Feel free to edit my answer to fix any remaining factual mistakes. Or add your own answer, if you prefer. $\endgroup$
    – Richard Hardy
    Commented Jun 15, 2017 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.