- I cross validate a lasso regression with multiple values of lambda (the multiplier for the penalty) e.g. from 0.00001 to 100
- I get the best solution is with a certain lambda, e.g. 0.7
- Given some of the coefficients have been zeroed, that best model is using a subset of features Fsub, e.g. 10% of the features are not used
- I run (and cross validate) a normal linear regression with Fsub (i.e. the same features lasso decided to use)
When I compare that best lasso regression with lambda 0.7 vs. the linear regression with Fsub, should I expect better results, same results, worse results (or depending on the case, all possible outcomes are possible)?
My feeling is that I can expect all possible results but I want to have a second opinion.
When I'm talking about better/same/worse I'm talking about the loss score. So, for instance, cross-validated MSE or RSS.