3
$\begingroup$

I'm reading Kirkland et al. "LASSO Tuning Parameter Selection" (2015) regarding methods for selecting the tuning Parameter in LASSO regressions. I'm a bit confused about the following Statements.

"This paper attempts to provide an overview of methods which are available to select the value of the tuning parameter for either prediction or variable selection purposes"

Why the tuning parameter should differ between the two purposes ?

I mean if we can select the "true" model, this should also result in the best forecast, or is my intuition wrong ?

$\endgroup$
5
  • 3
    $\begingroup$ Your intuition is reasonable, but turns out not to be quite right. Generally you want a smaller penalty for prediction accuracy, as it is a little better to err on the side of making the model too big. This manifests in the AIC/BIC debate: AIC is optimal for a certain type of prediction error but is not even consistent for variable selection. $\endgroup$
    – guy
    Commented May 27, 2018 at 22:05
  • $\begingroup$ I'm still not getting the right Intuition. May you can extend or comment or reffer to some sources, that explain the difference $\endgroup$
    – Leo96
    Commented May 27, 2018 at 22:30
  • 2
    $\begingroup$ If you are going to select the wrong predictors, would you rather select a set which is too big (has False Positives) or too small (has False Negatives)? If your goal is prediction, having a few FPs is not too big of a deal, but a FN is a disaster. Consequently, the optimal thing to choose a penalty which will have a few FPs but no FNs; this leads to inconsistent selection. On the other hand, if you value variable selection then you can treat FPs and FNs more equally; this leads to suboptimal prediction accuracy (because when you get an FN it is really bad) but you get consistent selection. $\endgroup$
    – guy
    Commented May 28, 2018 at 0:37
  • $\begingroup$ You may have erred in seeking parsimony, which is more often than not the enemy of predictive discrimination. But even worse, you did not validate the stability of the set of features selected by lasso. Do a little simulation or bootstrapping to see if the lasso agrees with itself in the list of features selected. If it doesn't you are just wasting your time and losing all interpretation ability. $\endgroup$ Commented May 28, 2018 at 11:39
  • $\begingroup$ Thanks guy, that helped a lot for intuition. Do you have maybe some sources or academic literature to that ? Btw: I can just vote your comment up $\endgroup$
    – Leo96
    Commented Jun 5, 2018 at 10:04

1 Answer 1

4
$\begingroup$

For prediction, we are interested in the best outcome and want to include as much information as possible in the model to explain the response, but still without over fitting (don't capture the noise) as we want our model to generalize well to new data. Generally, lower values of the LASSO tuning parameter are needed for prediction. When a group of correlated variables are present, LASSO will tend to select the best predictor in the group and discard the rest. The true set of variables are usually included in the best predictive model with high probability. Other variables will be included to maximize the predictive power of the model and will likely have small parameter estimates but won't be shrunk completely to zero. The best predictive model will have the lowest mean squared error (MSE).

Variable selection is a much harder problem than prediction. For variable selection we want to understand the relationships between variables and explain the importance of each variable. This would in fact be recovering the true model. A higher value of the tuning parameter is necessary in order to shrink more parameters to zero. LASSO only does well for variable selection under some rather strong assumptions regarding the size of the parameters and the correlations between variables (see my answer here). Without these assumptions, the bias is usually very high in the models. In that case, a two stage procedure like relaxed LASSO or adaptive LASSO are better suited.

There are models with the oracle property, i.e. they are consistent for prediction and variable selection, containing the correct subset of variables only and having lowest MSE. Think of it as fitting a least squares model where the set of true predictor variables known. Adaptive LASSO has the oracle property (see my answer here), as do some shrinkage methods with concave penalty functions like MCP and SCAD (see my answer here). Hope that clarifies the issue for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.