I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve.
I have successfully run both glmnet() and cv.glmnet() using the "swiss" data example, and the lambda x MSE plot looks normal (i.e., how they look in online code examples).
However, when I use my actual data, the lambda x MSE plot comes out as follows (and tends to be variations of the same trends regardless of whether predictors are standardized or what the value of alpha is):
This post suggests that one potential problem causing such a trend is that my predictors are lowly correlated with the criteria. In this particular case, several predictors are correlated r>.2 with the criteria. Yet, in that post, error increases as more predictors are added, whereas my data eventually begins to reduce error as a high number of predictors are added.
I'm particularly wondering if anyone can explain why the MSE would increase so drastically before then decreasing as more predictors are further added?