Why is the penalty term $R(f)$ added to a general loss function in regularization instead of subtracting?
For example, $$ \mathrm{argmin} \sum L(\theta,\hat\theta)+ \lambda R(f) ? $$
0
$\begingroup$
$\endgroup$
-
2$\begingroup$ Because you're attempting to minimize the loss function subject to a penalty. Hence the argmin. If you subtracted it then you could make your R(f) huge and it wouldn't act as a penalty. $\endgroup$ – ilanman Sep 25 '16 at 16:26
-
$\begingroup$ Thank you! @ilanman I now understood the math behind it! $\endgroup$ – Harini Devulapalli Sep 25 '16 at 16:49
-
3$\begingroup$ Because you want to impose a penalty for "bad behavior", not grant a bonus. $\endgroup$ – Mark L. Stone Sep 25 '16 at 17:26
-
$\begingroup$ You want both the loss and the the thing the regularization term penalizes (it might be a measure of roughness or complexity, for example) to be small. $\endgroup$ – Glen_b♦ Sep 26 '16 at 0:45
3
$\begingroup$
$\endgroup$
Let me start with the concept of Regularization. Regularization is means to avoid high variance in model (also known as overfitting). High variance means that your model is actually following all noise and errors in the data. The model is not at all flexible. Since the idea is to control complexity, we want to penalize the model for overfitting.
The parameters of a model are decided based on the cost function of the model. The best model will have minimum cost. Let me take the example of linear regularization.
Cost function and parameters(theta) of Linear Model without Regularization:
Cost function and parameters(theta) of Linear Model with Regularization:
So, by using regularization, the parameters are penalized for over fitting. (regularized term is subtracted from the parameter to minimize the cost function)
-
$\begingroup$ Note that you can use Latex typesetting here by putting equations between dollar signs eg
$x$produces $x$ $\endgroup$ – Silverfish Sep 25 '16 at 19:08
