L2 norm regularization penalizes large weights to avoid overfitting, basically by subtracting the magnitude of the weight vector (times a regularization parameter) from each weight during each update.
However, if the weights are negative, the weight vector (and therefore the L2 norm) could have a really large magnitude. Thus, subtracting by the L2 norm would make them even more negative.
Am I misunderstanding how L2 norm regularization works?
This is our objective function, composed of a loss function and a regularizer.
$$\mathcal O(w,x,y) = \mathcal L(w,x,y)+\mathcal R(w)$$
So $\mathcal R(w)=\|w\|_2^2=\sum{w_i^2}$ in the case of $\ell_2$ regularization.
Let's perform gradient-based minimization, i.e. we will update params based on the negative partial derivatives of the objective function.
In that case:
$${\partial \mathcal R\over \partial w_i} = 2w_i$$
If $w_i \lt 0$, that partial derivative is less than zero as well, and the converse is also true in the case that $w_i \gt 0$, where we get a positive partial derivative.
What does that mean? It means that, if $w_i$ is negative, you have to increase it to minimize the norm, and if $w_i$ is positive, you have to decrease it.
L2 regularization adds $w_i^2$ term to the loss function. In iterative approaches using gradients, we subtract the gradient of the loss function not the magnitude of the weight itself. And in the loss function, the regularization part's derivative with respect to $w_i$ is going to be ${d\over dw_i}(w_i^2)=2w_i$. Typically, this part is multiplied with a chosen $\lambda$ to decrease/increase the importance of the regularization.
When the weight is negative, it moves towards the positive direction, i.e. $$w_{new}=w_{old}-\dots-(2w_{old})$$ and if it is positive, it moves towards the negative direction. In either case, you move towards origin (with small enough $\lambda$).
