3
$\begingroup$

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?

$\endgroup$
  • 1
    $\begingroup$ Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1) $\endgroup$ – usεr11852 Mar 31 at 12:04
5
$\begingroup$

For regularization terms similar to $\left\|\theta\right\|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.

Error terms such as $\sum_i \left\|y_i - f_{\theta}(x_i)\right\|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $\theta = 0$, leads to a high error.

We balance these two forces by using a regularization parameter ($\lambda$) in a summation like $$\frac{1}{N}\sum_{i=1}^{N} \left\|y_i - f_{\theta}(x_i)\right\|_2^2 + \lambda\left\|\theta\right\|_2^2,$$ where higher $\lambda$ forces the model toward more simplicity.

$\endgroup$
  • $\begingroup$ So, regularization like L2, L1 correspond to the first case, right? $\endgroup$ – alienflow Mar 31 at 12:05
  • 1
    $\begingroup$ @alienflow yes they all force toward zero (most simple). $\endgroup$ – Esmailian Mar 31 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.