1
$\begingroup$

Preface: I am aware of this post: Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression?

(It is generally recommended to use lambda.min or preferably lambda.1se).

However, if I pick lambda.min, all predictors remain in my model; if I pick lambda.1se, all predictors are dropped from model.

When I go for a linear model with all variables (lambda.min variant), several predictors seem to be uninformative (no significant relevance for model).

Edit: Conducting a OLS-regression seems to be a no-go in this case - I understand the rationale. However, I wonder, how I can assess model quality apart from predictive power in LASSO-setting?

Since lambda.1se seems to be a convention, I wondered whether it is possible to pick something in between, like lambda.0.5se (lambda.1se/2). I tried it out and it seems to be more informative within variable selection (some predictors remain in model, some predictors are dropped). Is this a reasonable approach?

Edit: I added a graph with lambda/MSE for more information (thanks for hint, @StupidWolf). I guess that it tells me that there is no suitable lambda for a really low CV error, right?

Lambda/MSE

Data set contains around 250 rows, 9 predictor variables, 1 continuous outcome variable. Any advice for me?

$\endgroup$
4
  • 1
    $\begingroup$ this is quite drastic. Can you share the plot for lambda vs loss / rmse ? I am referring to the output from plotting the cvfit object e.g cvfit = cv.glmnet(x, y) ; plot(cvfit) $\endgroup$ – StupidWolf May 24 '20 at 22:45
  • 1
    $\begingroup$ What do you mean by "When I go for a linear model with all variables (lambda.min variant)..."? Does that mean you are taking the predictors selected by LASSO and using them in a standard, non-penalized regression? That's generally not a good idea; see discussion on this page. See this page for more discussion about the 1se convention. $\endgroup$ – EdM May 24 '20 at 22:49
  • $\begingroup$ @StupidWolf I added the requested information. Could you tell me what you think about it? Thanks a lot! $\endgroup$ – Jens May 25 '20 at 7:01
  • $\begingroup$ @EdM Thanks for your input. Seems logical. However, how can I assess model quality apart from its predictive power? I am trying LASSO because I don't want to fall back to conventional stepwise backward selection procedures.. $\endgroup$ – Jens May 25 '20 at 7:02
2
$\begingroup$

For this data set, it seems that no choice of $\lambda$ will do very well. I would argue that LASSO isn't very helpful here. The problem is that the predictors don't seem to predict very well, at least in the way you have modeled them.

With 250 cases and 9 predictors you have over a 25/1 ratio of cases to predictors. In most circumstances that should be more than enough to do ordinary least squares without a need for the penalization and variable selection provided by LASSO.

The curve of mean-squared error (MSE) versus $\lambda$ makes that pretty clear. At the minimum-MSE $\lambda$ value, the axis labels along the top show that all 9 predictors are included in the model! So you're not getting variable selection. And the cross-validated MSE isn't that much lower than what the essentially unpenalized models at very low $\lambda$ values provide. So in this model LASSO does almost nothing useful.

Instead of worrying about the best $\lambda$ value to choose, consider whether the way you are modeling your data is missing something important: a non-linearity in some of the predictors, or interaction terms that, if incorporated into the model, would provide better predictive power. Depending on what you are trying to accomplish, an approach like boosted regression trees (which can incorporate interactions to whatever level you like) might be a better choice--if your predictors are in fact related to your outcome at all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.