I am performing Lasso Regularization on my linear model. When I steadily increase my lambda in cross-validation. I see the residual sum of squares will increasing steadily in my training set. Since Lasso adding more and more penalty which minimizes the weights/parameters of the linear model.

What about for my test set? What kind of RSS change would I expect to see on my test data if I increase my lambda from 0? and Why?

  • $\begingroup$ It may decrease if model without regularization overfitted the training set, and, if I'm not mistaken, will converge to $TSS$ (Total sum of squares) as $\lambda \to \infty$. $\endgroup$ – Łukasz Grad Feb 27 '17 at 20:10
  • $\begingroup$ Is there any mathematical explanation that I can take a look? $\endgroup$ – Ahmet Salih Gundogdu Feb 27 '17 at 20:19
  • $\begingroup$ Regarding convergence to $TSS$? For $\lambda \to \infty$ regularized regression simplifies to model with intercept only $$Y = \beta_0 + \epsilon$$. And you can check that for this model $$\hat{\beta}_0 = \bar{Y} = \frac{1}{n}\sum_i Y_i$$ $\endgroup$ – Łukasz Grad Feb 27 '17 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.