We often avoid overfitting by penalizing the norm of the weights/coefficients (in a classic Ridge or Lasso regression). I understand that we want smooth functions as they will be more likely to generalize. But what is the link between smoothness and norm?

  • $\begingroup$ I wanted to write weight's norm obviously. $\endgroup$ – HumanLearner Apr 8 at 14:14
  • $\begingroup$ I don't understand what you mean by "link", are you looking for a theoretical explanation of penalized regression or something more practical? $\endgroup$ – RScrlli Apr 8 at 19:32
  • $\begingroup$ Thank you for you answer. I am looking for a theoretical explanation. Why penalizing high coefficients leads to smoother functions ? $\endgroup$ – HumanLearner Apr 9 at 8:30
  • $\begingroup$ As I see it, the penalisation force the coefficients to shrink, in the case of Lasso actually some coefficients become zero. The smoothness is a consequence of shrinking the coefficients toward zero. $\endgroup$ – RScrlli Apr 9 at 8:49
  • $\begingroup$ For instance LASSO can be used as a variable selection model, and it's obvious that by reducing the complexity of your model i.e. by reducing the number of covariates the over-fitting will be reduced and hence the approximation will be smoother. (The smoothest function you could imagine is a straight horizontal line where you include only an intercept). $\endgroup$ – RScrlli Apr 9 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.