# Does regularization penalize models that are simpler than needed?

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?

• Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1) – usεr11852 Mar 31 at 12:04

For regularization terms similar to $$\left\|\theta\right\|_2^2$$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.
Error terms such as $$\sum_i \left\|y_i - f_{\theta}(x_i)\right\|_2^2$$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $$\theta = 0$$, leads to a high error.
We balance these two forces by using a regularization parameter ($$\lambda$$) in a summation like $$\frac{1}{N}\sum_{i=1}^{N} \left\|y_i - f_{\theta}(x_i)\right\|_2^2 + \lambda\left\|\theta\right\|_2^2,$$ where higher $$\lambda$$ forces the model toward more simplicity.