Picking lambda for LASSO

Question

Preface: I am aware of this post: Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression?

(It is generally recommended to use lambda.min or preferably lambda.1se).

However, if I pick lambda.min, all predictors remain in my model; if I pick lambda.1se, all predictors are dropped from model.

When I go for a linear model with all variables (lambda.min variant), several predictors seem to be uninformative (no significant relevance for model).

Edit: Conducting a OLS-regression seems to be a no-go in this case - I understand the rationale. However, I wonder, how I can assess model quality apart from predictive power in LASSO-setting?

Since lambda.1se seems to be a convention, I wondered whether it is possible to pick something in between, like lambda.0.5se (lambda.1se/2). I tried it out and it seems to be more informative within variable selection (some predictors remain in model, some predictors are dropped). Is this a reasonable approach?

Edit: I added a graph with lambda/MSE for more information (thanks for hint, @StupidWolf). I guess that it tells me that there is no suitable lambda for a really low CV error, right?

Data set contains around 250 rows, 9 predictor variables, 1 continuous outcome variable. Any advice for me?

Answer 1

For this data set, it seems that no choice of $\lambda$ will do very well. I would argue that LASSO isn't very helpful here. The problem is that the predictors don't seem to predict very well, at least in the way you have modeled them.

With 250 cases and 9 predictors you have over a 25/1 ratio of cases to predictors. In most circumstances that should be more than enough to do ordinary least squares without a need for the penalization and variable selection provided by LASSO.

The curve of mean-squared error (MSE) versus $\lambda$ makes that pretty clear. At the minimum-MSE $\lambda$ value, the axis labels along the top show that all 9 predictors are included in the model! So you're not getting variable selection. And the cross-validated MSE isn't that much lower than what the essentially unpenalized models at very low $\lambda$ values provide. So in this model LASSO does almost nothing useful.

Instead of worrying about the best $\lambda$ value to choose, consider whether the way you are modeling your data is missing something important: a non-linearity in some of the predictors, or interaction terms that, if incorporated into the model, would provide better predictive power. Depending on what you are trying to accomplish, an approach like boosted regression trees (which can incorporate interactions to whatever level you like) might be a better choice--if your predictors are in fact related to your outcome at all.