Backpropagation - partial derivatives

Question

I am currently reading the Neural networks and deep learning book by Michael Nielsen.

I have a question regarding the backpropagation chapter:

Background:

He explains the influence of a neuron on the cost function by saying that there sits a demon in the neuron. The demon sits at the $j^{th}$ neuron in layer $l$. As the input to the neuron comes in, the demon messes with the neuron's operation. It adds a little change $\Delta z_j^l$ to the neuron's weighted input, so that instead of outputting $\sigma(z_j^l)$, the neuron instead outputs $\sigma (z_j^l+\Delta z_j^l)$. This change propagates through later layers in the network, finally causing the overall cost to change by an amount $\frac{\partial C}{\partial z_j^l} \Delta z_j^l$.

Furthermore, he states that if $\frac{\partial C}{\partial z_j^l}$ has a large value (either positive or negative). Then the demon can lower the cost quite a bit by choosing $\Delta z_j^l$ to have the opposite sign to $\frac{\partial C}{\partial z_j^l}$. By contrast, if $\frac{\partial C}{\partial z_j^l}$ is close to zero, then the demon can't improve the cost much at all by perturbing the weighted input $z_j^l$.

So far, I understand this. $\frac{\partial C}{\partial z_j^l}$ quantifies the influence of $z_j^l$ on the cost function $C$.

Question:

However, afterwards he states that there is a heuristic sense in which $\frac{\partial C}{\partial z_j^l}$ is a measure of the error in the neuron.

Why is $\frac{\partial C}{\partial z_j^l}$ a measure of error in the neuron?

I thought that it is just the influence of the neuron output on the cost function.

• Wouldn't that imply that influence equals error?

• Aren't there cases in which we want certain neurons to have greater influence on the cost function?

Answer 1

The expression $\frac{\partial C}{\partial z_j^l}$ represents how much does the change of the weight $j$ in the layer $l$ influence the cost $C$. You are right that this is not equal to the absolute error caused by that weight¹.

However, in neural networks, empirically there seems to be a relationship between the magnitude of the derivative and the error: Around the (local) optimum, the surface will be rather flat and will be steeper further away.

To answer the first part of your question: They are not equal, but it is a convenient approximation.

To answer the second part: In a typical neural network, the neurons have no predefined meaning, so we have no information which of them should be important and which shouldn't. Moreover, the aim of the training is to get the weights into a state where all the derivatives are close to zero. That means we want any local change in the weights to have zero influence on the cost.

Finally, for completeness, I am including the picture of the demon in question:

---

¹ For example, take the $f(x) = |x|$: This function has the same derivative everywhere (1 or -1 on the positive and negative part, respectively), regardless of how far from the minimum you are.