# Why is the penalty term added instead of subtracting it from loss term in regularization?

Why is the penalty term $R(f)$ added to a general loss function in regularization instead of subtracting?

For example, $$\mathrm{argmin} \sum L(\theta,\hat\theta)+ \lambda R(f) ?$$

• Because you're attempting to minimize the loss function subject to a penalty. Hence the argmin. If you subtracted it then you could make your R(f) huge and it wouldn't act as a penalty. Sep 25 '16 at 16:26
• Thank you! @ilanman I now understood the math behind it! Sep 25 '16 at 16:49
• Because you want to impose a penalty for "bad behavior", not grant a bonus. Sep 25 '16 at 17:26
• You want both the loss and the the thing the regularization term penalizes (it might be a measure of roughness or complexity, for example) to be small. Sep 26 '16 at 0:45

• Note that you can use Latex typesetting here by putting equations between dollar signs eg $x$ produces $x$ Sep 25 '16 at 19:08