4
$\begingroup$

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?

$\endgroup$
1
  • 2
    $\begingroup$ Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1) $\endgroup$
    – usεr11852
    Mar 31, 2019 at 12:04

1 Answer 1

6
$\begingroup$

For regularization terms similar to $\left\|\theta\right\|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.

Error terms such as $\sum_i \left\|y_i - f_{\theta}(x_i)\right\|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $\theta = 0$, leads to a high error.

We balance these two forces by using a regularization parameter ($\lambda$) in a summation like $$\frac{1}{N}\sum_{i=1}^{N} \left\|y_i - f_{\theta}(x_i)\right\|_2^2 + \lambda\left\|\theta\right\|_2^2,$$ where higher $\lambda$ forces the model toward more simplicity.

$\endgroup$
2
  • $\begingroup$ So, regularization like L2, L1 correspond to the first case, right? $\endgroup$
    – alienflow
    Mar 31, 2019 at 12:05
  • 1
    $\begingroup$ @alienflow yes they all force toward zero (most simple). $\endgroup$
    – Esmailian
    Mar 31, 2019 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.