How does L2 penalize large weights

Question

The L2 regularization term is useful because it penalizes large weights over smaller weights which is good to prevent overfitting. I'm having a hard time understanding how exactly it does this.

This is what I know about L2. I know that one can take a standard cost function and use the following statistical approach using a prior distribution...

$$P(\Theta \mid x) = \frac{P(x\mid\Theta )P(\Theta )}{P(x)}$$

and use the MAP rule...

$$\operatorname{argmax}\sum log(P(x_i\mid\Theta )P(\Theta)$$

to re-write the cost function in the following format...

$$\log(P(x_i\mid\Theta)) + \log(P(\Theta))$$

where $\log(P(\Theta))$ is essentially the regularization term. My question is, how is this prior, $\log(P(\Theta))$, penalizing large weights? It is not clear to me. How does the prior (regularization term) penalize large weights and generalize to prevent overfitting?

Perhaps my understanding of priors is pretty basic, which might be why I can't see how this is done, I'm hoping someone can explain it to me via examples or proofs.

Answer 1

In $L_2$ regularization, your log-prior, $\log P(\Theta)$, corresponds to a term like $\lambda \sum \theta_i^2$ in the cost function that we want to minimize, in which it is equivalent to maximizing the negative of the cost function: $-(...)-\lambda\sum \theta_i^2$, like we do in MAP (i.e. maximizing some objective). There, the regularization part is assumed to be coming from the prior, i.e. $$\log P(\Theta)= -\lambda\sum \theta_i^2+C\rightarrow P(\Theta)\propto \exp \left(-\lambda\sum \theta_i^2\right)$$

Here, $P(\Theta)$ is proportional to an expression which is in Gaussian form, i.e. with a proper normalizing constant the RHS expression is a Gaussian PDF with zero-mean and $\frac{1}{2\lambda}$ variance.

So, you have a prior belief on your parameters which says that they're zero-mean, and with suitably large $\lambda$, and small variance. So, if you have large values for the parameters, the value coming from your prior PDF will be small and you won't have much chance to maximize your MAP.