I have a data set with 2 features and I'm trying to predict one real-valued variable. I use linear regression and I measure the error using 10-fold CV and absolute mean error as a metric. I noticed that I get a smaller error when I predict only off one feature, so I assumed it may be because the other one introduces noise. However, I decided to replace linear regression with L1 regularized regression (Lasso) on both variables, and I was hoping that varying the complexity parameter (alpha), I should get results at least as good as linear regression with only one feature, because if that's the best case, Lasso would just make the weight associated with that variable 0. This was not the case however, and Lasso performs worse on all parameters I've tried. Can anyone suggest some ideas why this may be the case?
FWIW, I tried Ridge too and it performs slightly better than Lasso, but still worse than LR with no regularization (using sklearn implementations). Also, to be clear, both L1 and L2 perform better than vanilla LR on all features, but not better than LR on just one feature.