I have a data set with 2 features and I'm trying to predict one real-valued variable. I use linear regression and I measure the error using 10-fold CV and absolute mean error as a metric. I noticed that I get a smaller error when I predict only off one feature, so I assumed it may be because the other one introduces noise. However, I decided to replace linear regression with L1 regularized regression (Lasso) on both variables, and I was hoping that varying the complexity parameter (alpha), I should get results at least as good as linear regression with only one feature, because if that's the best case, Lasso would just make the weight associated with that variable 0. This was not the case however, and Lasso performs worse on all parameters I've tried. Can anyone suggest some ideas why this may be the case?

FWIW, I tried Ridge too and it performs slightly better than Lasso, but still worse than LR with no regularization (using sklearn implementations). Also, to be clear, both L1 and L2 perform better than vanilla LR on all features, but not better than LR on just one feature.

  • $\begingroup$ Do you mean that the cross-validation performance as a function of $\lambda$ has maximum at $\lambda=0$ and is decreasing all the way from it? Can you show the curve? $\endgroup$ – amoeba Apr 22 '16 at 13:20
  • 1
    $\begingroup$ The idea of penalization is to control for overfitting (or rather, regularize the inverse matrix). If you have few features and they are not collinear, what is the point of penalizing? $\endgroup$ – dv_bn Apr 22 '16 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.