In machine learning, it is often advised to use weight regularization so that the model parameters don't grow big while training. I am not convinced that having small weights improves the model's ability to generalize. Why should a model have small weights? What happens when a model has 'big' weights? I would really appreciate it if you could give a simple example to illustrate your answer!
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Regularization is used to reduce variance in the model in order to avoid overfitting. If you do not use regularization, your coefficients will take extreme values in order to perfectly fit your datapoints, but will completely fail to make accurate predictions for new data. I believe that is why you want your coefficients to take more moderate values.
I am fairly new to machine learning, but I have these two references from when I started learning:^ https://towardsdatascience.com/regularization-an-important-concept-in-machine-learning-5891628907ea https://towardsdatascience.com/regularization-in-machine-learning-76441ddcf99a
Hope it helped!
