If you have stochastic gradient descent and you want to update the weights $w_t$ with the L2 norm regularization penalty, how is this math done? I am using sklearn for this and am just wondering how the weight $w_m$ is updated according to $l- \lambda||w||_{2}^{2}$. I read through the sklearn docs but they didn't go into to much detail about how this is actually performed mathematically.


It's the same procedure as SGD with any other loss function. The only difference is that the loss function now has a penalty term added for $\ell_2$ regularization.

The standard SGD iteration for loss function $L(w)$ and step size $\alpha$ is: $$w_{t+1} = w_t - \alpha \nabla_w L(w_t)$$

Say the original loss function was $L_0$, and you add a penalty term for $\ell_2$ regularization. The new loss function is: $$L(w) = L_0(w) + \lambda \|w\|_2^2$$

The gradient is: $$\nabla_w L(w) = \nabla_w \left [ L_0(w) + \lambda \|w\|_2^2 \right ] = \nabla_w L_0(w) + 2 \lambda w$$

Plug this into the SGD iteration above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.