I am going through online deep learning book and trying to recreate Neural Network that was written there with a bit of different class designs. However, I've run into a problem, where when using L2 Regularization I can't see its impact on backpropogation formulas. Here's what I mean. The only formula in backpropogation that uses loss function derivative is the one defining error for output layer and is defined as follow:
error = C'(a) * a'(z)
where C'(a) is loss function derivative with respect to activation and a'(z) is activation function derivative with respect to weighed input. I don't see how any part of this equation changes when adding L2 regularization. I believe it should be derivative of loss function with respect to activation that should change, however we're only adding squared weights which should disappear when calculating derivative(since it is with respect to activation, not weights). Something should be wrong with my logic, please tell what it is.
EDIT: To be more specific. Suppose we use Quadratic Loss function with L2 regularization. Is the follow true and if not, why?
C'(a) = a - y
where a is activation and y is desired output.