0
$\begingroup$

I am going through online deep learning book and trying to recreate Neural Network that was written there with a bit of different class designs. However, I've run into a problem, where when using L2 Regularization I can't see its impact on backpropogation formulas. Here's what I mean. The only formula in backpropogation that uses loss function derivative is the one defining error for output layer and is defined as follow:

error = C'(a) * a'(z)

where C'(a) is loss function derivative with respect to activation and a'(z) is activation function derivative with respect to weighed input. I don't see how any part of this equation changes when adding L2 regularization. I believe it should be derivative of loss function with respect to activation that should change, however we're only adding squared weights which should disappear when calculating derivative(since it is with respect to activation, not weights). Something should be wrong with my logic, please tell what it is.

EDIT: To be more specific. Suppose we use Quadratic Loss function with L2 regularization. Is the follow true and if not, why?

C'(a) = a - y

where a is activation and y is desired output.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For your cost function, if you use L2 regularization, besides the regular loss function, you need add additional loss caused by high weights. Basically you need to add the below value to your loss function. Lambda is a hyperparameter controls the L2 regularization. When it equals 0, it is like no regularization at all. m is the number of instances.

enter image description here

Now when you do back propagation and calculate the derivative, you need to calcuate this additional cost's derivative too

enter image description here

When you update the weights, you need to substract the learning rate * the additional derivative. Thus it pushes the weight even lower, which is called weight decay.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you, this approach would work, however, I don't want to add any unnecessary dependencies on optimizer, I'd like that only loss function objects had dependencies on regularization. This is why I asked how would I find derivative of the new loss function(with regularization) with respect to output activations and how would it change. $\endgroup$
    – Виталик Бушаев
    Commented Sep 12, 2017 at 15:41
  • $\begingroup$ Ultimately, you are trying to calculate the derivate of the loss with respect to the weights, right? That can be calculated to be the derivative of loss with respect to the output of activation * derivative of activation with respect to the output of the linear function (W*X+B) * derivative of the linear function with respect to weights. $\endgroup$
    – Lan
    Commented Sep 12, 2017 at 16:08
  • $\begingroup$ exactly, so that's the question, how would derivative of the loss function with respect to output of activations change after adding L2 ? Because other derivatives don't change, that's the only one that does, but i can't see how regularization would impact that first derivative. $\endgroup$
    – Виталик Бушаев
    Commented Sep 12, 2017 at 16:29
  • $\begingroup$ IT IS CHANGED. Because of the loss function has the additional regularization part (my first pic), its derivative with respect to activation has an additional part too. The derivative of the regularization part does not disappear. $\endgroup$
    – Lan
    Commented Sep 12, 2017 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.