6
$\begingroup$

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve.

I have successfully run both glmnet() and cv.glmnet() using the "swiss" data example, and the lambda x MSE plot looks normal (i.e., how they look in online code examples).

However, when I use my actual data, the lambda x MSE plot comes out as follows (and tends to be variations of the same trends regardless of whether predictors are standardized or what the value of alpha is):

MSE as function of lambda and number of predictors

This post suggests that one potential problem causing such a trend is that my predictors are lowly correlated with the criteria. In this particular case, several predictors are correlated r>.2 with the criteria. Yet, in that post, error increases as more predictors are added, whereas my data eventually begins to reduce error as a high number of predictors are added.

I'm particularly wondering if anyone can explain why the MSE would increase so drastically before then decreasing as more predictors are further added?

Here's a similar looking plot with alpha=1 (i.e., LASSO): enter image description here

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Is this the cross-validated MSE or the MSE on the training data (i.e. the data used to estimate the regression coefficients)? And what do the error bars mean? $\endgroup$
    – Ruben van Bergen
    Jul 7, 2020 at 9:21

1 Answer 1

Reset to default
1
$\begingroup$

This can be explained by the trade of between bias and variance. We know that mse = bias^2 + var, a sum of a decreasing function of the number of predictors involved (bias) and an increasing function (var).

The thing, we dont have a specific role about the behavior of mse (training mse), but generally it decreased with more predictors involved, but its not always the case.

In this situation your mse movement is justified by the fact bias is slowly decreasing compared to how variance is going up (the first part of the graph ) . And when it reaches its max value of complexity(var), any predictor added will only decrease the biais so the mse will go down.

There may be other explanations that comes out of your data set. For example a large variance of the target. If so your model var will take more predictors until it reaches the max mse before starts decreasing

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.