Specifically, why does Equation (BP1) not have the same form as (29)?1

In order to explain backpropagating the gradient through the neural net, he starts by defining something he calls error (Equation (29)).

nielsen error

The error in a neuron j in a layer l is given by the partial derivative of the cost function C with respect to the weighted input z of the neuron j in layer l. Weighted input to a neuron is simply the input to that neuron, multiplied by its weight, summed with the neuron's bias. It is what the neuron's activation function takes as its input.

He then defines the error for a neuron in the final, output layer of the net as follows (Equation (BP1)): nielsen error output layer

Where $\sigma$ is the activation function and $\sigma'$ is its derivative.

What I thought he was going to do was use the first definition of error in combination with the chain rule to derive $\partial C$ in terms of $\partial w$ and $\partial b$ for the j-th neuron in layer l, thereby providing a way to backpropagate through the net.

Why does he change the definition for error immediately after he defines it? Are these two equations actually saying the same thing? If so why?

1 Nielsen, Neural Networks and Deep Learning, Chapter 2 - http://neuralnetworksanddeeplearning.com/chap2.html


Okay bit of a goof on my part. Nielsen addresses this a couple sections later in the same chapter, in Proof of the four fundamental equations (optional).

Briefly, Eqn (29) above can be re-expressed (using the chain rule) as partial derivatives with respect to the output activations aL. Simplifying gives you the form seen in (BP1).

| cite | improve this answer | |

In Nielsens explanation of bp, he doesn't go in to the details and actually complicates it. There is a video by 3blue1brown on youtube that explains bp really well by simplifying the case to just one neuron for each layer and then when you add more neurons it is not that different.

Highly recommend watching that 7 minute video and then coming back to Nielsens for the code.

| cite | improve this answer | |
  • 2
    $\begingroup$ Could you please add a link to that video? $\endgroup$ – kjetil b halvorsen Dec 5 '19 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.