Assume I have 10 features to predict an outcome and I use LASSO regression. Let's say the RMSE of the test set is 20.
Now, I introduce 5 more features and predict the same outcome, and I also use LASSO. The RMSE of the test set is getting larger, to 80.

Would it be possible? Why would it happen?

LASSO shrinks the parameters to 0. And if the new features could not improve the prediction ability, why would it not shrink the new features to 0 making the new model become the original model? In terms of this, wouldn't it produce a similar RMSE for the test set, not a very different one?

  • $\begingroup$ You are standardizing all your predictors, right? $\endgroup$ – TBSRounder May 17 '16 at 13:12
  • $\begingroup$ no I didn't. The predictors are count, all non-negative value. $\endgroup$ – Lazar May 17 '16 at 14:26
  • 2
    $\begingroup$ ok good. What about the tuning? I would expect your RMSE to go down in your training set after adding the variables, but not necessarily in your test set. $\endgroup$ – TBSRounder May 17 '16 at 15:39
  • 3
    $\begingroup$ It might help if you described the exact procedure you used to train the models (it sounds like you're using glmnet, so the relevant code would be helpful). The situation you describe sounds like you're overfitting to the training set. As TBSRounder pointed out, cross-validation is a good approach to avoid overfitting. $\endgroup$ – josliber May 18 '16 at 19:07
  • 1
    $\begingroup$ @RichardHardy Not necessarily true. When it comes to the "Data reduction" of penalized regression, LASSO will tend to pick a feature that tends to be representative of many collinear variables whereas Ridge will give an efficient representation of their joint effect using a linear combination of all their effects. It may be compared to selecting a feature with high loadings on the first principal component (LASSO) or generating the first orthonormal basis (RIDGE) in unsupervised learning. $\endgroup$ – AdamO May 23 '16 at 18:50

It's true that LASSO encourages sparseness in the sense that $\beta$s which are very close to zero are set to zero exactly. So if the tendency is to maintain features which have large values of $\beta$ this may not lead to better prediction in the sense of RMSE.

A $\beta$ for a newly introduced feature may be very large because that feature has a low prevalence or low variability, so it enhances prediction in a small group of observations that is very different at the sacrifice of loosing predictive accuracy among the masses who were discriminated better by smaller $\beta$s

As an example here is a trivariate relationship between a continuous feature $x$ and a binary feature $w$ with an outcome $Y$. The $x$ effect explains much more of the variability in these data than the $w$ effect despite the $x$ effect being smaller overall than the $w$ effect, even after standardization. Lasso would favor $w$ in a model over $x$ because of its magnitude, but that alone does not suffice to lead to good prediction, we merely select $w$ because it is good at discriminating participants. This is the type of feature LASSO tends to select.

enter image description here

This underscores the importance of using cross-validation to select the tuning parameter in a LASSO model. In one such as this, you would find a much better predictive accuracy by including both effects.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.