1
$\begingroup$

I do a rige and lasso regression on a train data set and get the lambdas via cross validation and evalute the prediction accuracy on a test data set.

After that i do the same procedure for the same data but adding polynomials to the power of 4 and interactions to the order of 2 (V1*V2)+(V1*V3) to the train and test data set.

At the end i get a smaller test mse with ridge for the model with interactions and polynomials compared to the model without interactions. That's a result that i would expect.

But for lasso I have a higher test mse compared to the model without interactions and polynomials. That's what i don't expected.

I don't understand why lasso performs worse than ridge and worse than a model with less explanatory variables?

$\endgroup$
  • $\begingroup$ Please don't open a new question. Instead, edit the existing one. $\endgroup$ – Stephan Kolassa Jul 16 '18 at 15:54
  • $\begingroup$ @StephanKolassa its a different question. The first one is about ols compared to ridge and lasso. This one is ridge compared to lasso and . And i also use different data setup $\endgroup$ – Dima Ku Jul 16 '18 at 15:56
  • 1
    $\begingroup$ Thank you for clarifying. However, given that the two questions are very closely related, please include links next time. $\endgroup$ – Stephan Kolassa Jul 16 '18 at 15:58
  • $\begingroup$ This sounds like an implementation issue. Are you using Elastic Net? $\endgroup$ – ERT Jul 16 '18 at 16:17
  • $\begingroup$ no i just used ridge and lasso $\endgroup$ – Dima Ku Jul 16 '18 at 16:32
1
$\begingroup$

As Frank Harrell notes in an answer to another question, ridge generally performs better than LASSO in predictions. The main exception is if there really are only a handful of true predictors. In the real world with often correlated predictors, that frequently is not the case.

Your use of interactions and polynomials probably made this problem worse. Typical LASSO formulations do not try to keep main effects and their related interaction terms together in models, so it's possible to develop models that include interaction terms without the associated main effects. That's often not a good idea.

If you also have polynomial modeling of the predictors involved in the interaction terms, then you are starting with a very large set of potential predictors. Ridge regression will include all of those predictors with appropriate penalization of the associated coefficients. LASSO, by its nature, will just choose a few in a way that might not be repeated from sample to sample.

Repeating LASSO selection of predictors on multiple bootstrap samples of your original data can be instructive. You may find wildly different sets of predictors chosen by LASSO among such samples. Ridge regression will keep all of the predictors, and you may be surprised to find how well the values of the penalized coefficients agree among such samples.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Why is having interaction terms with no main effect "often not a good idea" in the context of predictive models? $\endgroup$ – Matthew Drury Jul 16 '18 at 18:03
  • 2
    $\begingroup$ @MatthewDrury this is certainly more of an issue for inference than for prediction. Harrell's experience leads him to recommend keeping lower order effects in models when there are interactions. There are issues with lack of invariance to location shifts in continuous predictors and with coding of categorical predictors; see answers and links on this page. These problems will be exacerbated by including two 2-way interactions among predictors modeled as 4th-order polynomials, as in this question. $\endgroup$ – EdM Jul 16 '18 at 18:36
  • $\begingroup$ but i have interaction polynomials and the main effects . All is included $\endgroup$ – Dima Ku Jul 17 '18 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.