5
$\begingroup$

L2 norm regularization penalizes large weights to avoid overfitting, basically by subtracting the magnitude of the weight vector (times a regularization parameter) from each weight during each update.

However, if the weights are negative, the weight vector (and therefore the L2 norm) could have a really large magnitude. Thus, subtracting by the L2 norm would make them even more negative.

Am I misunderstanding how L2 norm regularization works?

$\endgroup$

2 Answers 2

7
$\begingroup$

This is our objective function, composed of a loss function and a regularizer.

$$\mathcal O(w,x,y) = \mathcal L(w,x,y)+\mathcal R(w)$$

So $\mathcal R(w)=\|w\|_2^2=\sum{w_i^2}$ in the case of $\ell_2$ regularization.

Let's perform gradient-based minimization, i.e. we will update params based on the negative partial derivatives of the objective function.

In that case:

$${\partial \mathcal R\over \partial w_i} = 2w_i$$

If $w_i \lt 0$, that partial derivative is less than zero as well, and the converse is also true in the case that $w_i \gt 0$, where we get a positive partial derivative.

What does that mean? It means that, if $w_i$ is negative, you have to increase it to minimize the norm, and if $w_i$ is positive, you have to decrease it.

$\endgroup$
1
  • $\begingroup$ Thanks! That makes sense! $\endgroup$
    – Benitok
    Feb 2, 2020 at 23:34
2
$\begingroup$

L2 regularization adds $w_i^2$ term to the loss function. In iterative approaches using gradients, we subtract the gradient of the loss function not the magnitude of the weight itself. And in the loss function, the regularization part's derivative with respect to $w_i$ is going to be ${d\over dw_i}(w_i^2)=2w_i$. Typically, this part is multiplied with a chosen $\lambda$ to decrease/increase the importance of the regularization.

When the weight is negative, it moves towards the positive direction, i.e. $$w_{new}=w_{old}-\dots-(2w_{old})$$ and if it is positive, it moves towards the negative direction. In either case, you move towards origin (with small enough $\lambda$).

$\endgroup$
3
  • 1
    $\begingroup$ Thanks! That makes sense! (sorry, saw Firebug's answer first so accepted his) $\endgroup$
    – Benitok
    Feb 2, 2020 at 23:35
  • $\begingroup$ @Benitok you can accept this answer instead of mine if you think it was more helpful :) $\endgroup$
    – Firebug
    Feb 3, 2020 at 0:45
  • $\begingroup$ It’s cool, I think we pretty much said the same thing. $\endgroup$
    – gunes
    Feb 3, 2020 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.