1. I cross validate a lasso regression with multiple values of lambda (the multiplier for the penalty) e.g. from 0.00001 to 100
  2. I get the best solution is with a certain lambda, e.g. 0.7
  3. Given some of the coefficients have been zeroed, that best model is using a subset of features Fsub, e.g. 10% of the features are not used
  4. I run (and cross validate) a normal linear regression with Fsub (i.e. the same features lasso decided to use)

When I compare that best lasso regression with lambda 0.7 vs. the linear regression with Fsub, should I expect better results, same results, worse results (or depending on the case, all possible outcomes are possible)?

My feeling is that I can expect all possible results but I want to have a second opinion.

When I'm talking about better/same/worse I'm talking about the loss score. So, for instance, cross-validated MSE or RSS.

  • 1
    $\begingroup$ What do you mean by "better" results? $\endgroup$ – Noah Oct 4 '18 at 4:01
  • 1
    $\begingroup$ I agree with Noah's statement here. What is your definition of a "better result"? Even if you, somehow, have selected the correct features with the Lasso step (not a guarantee by any means!) then the Lasso estimates should be biased due to the shrinkage. However, I am unsure if the OLS estimates are unbiased since they are conditional on the Lasso procedure finding them to be non-zero. My intuition is that they too would be biased due to this, but I am unsure. $\endgroup$ – Phil Oct 4 '18 at 6:54
  • 1
    $\begingroup$ You cannot compare easily like this, different algorithms will give different results. $\endgroup$ – user2974951 Oct 4 '18 at 7:18
  • $\begingroup$ Noah thank you for pointing out that "better" can be misleading. I just edited the post by adding my definition of better/same/worse in this context. $\endgroup$ – Bustic01 Oct 8 '18 at 1:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.