I'm attempting the final problem in Chapter 2 of Michael Nielsen's Neural Networks and Deep Learning book. The network in question is a simple 3-layer feedforward neural network with sigmoid activation functions for the neurons in the hidden and output layers.

When calculating the gradient, it appears I need to manage derivatives of the cost function with respect to each weight and bias parameter separately for each training example, for each stage of backpropagation until the final step; this leads to some rather ugly code. Is there any way to get around this?

  • $\begingroup$ I asked a similar question, and somebody answered basically "yes, you're stuck". I'm not entirely confident in the answer however... stats.stackexchange.com/questions/240971/… $\endgroup$ – generic_user Dec 12 '16 at 19:19
  • $\begingroup$ There is a trick for turning the bias into a feature and then you don't have to treat it differently than any of the weights. Is that the type of thing you're looking for? $\endgroup$ – Aaron Dec 12 '16 at 23:32
  • $\begingroup$ No, the issue is that I need to contend with a 3D array of weights for each connected pair of layers in the neural network, and it seems I can't sum out the third dimension in these arrays until the very end of the backpropagation process. The same problem occurs whether or not I treat the bias term as a "weight". $\endgroup$ – Jason Dec 13 '16 at 0:33

Within an iteration, you usually have to store the derivatives because you'll use them to update the weights after backpropagation. Especially some optimization algorithms (e.g. momentum, ADAM) also need the derivatives to maintain some internal status.

If at some point you can be sure that some derivatives won't be involved in the rest of the iteration you can erase them immediately to save some memory, though I don't think this will make the code simpler.

You don't have to keep the derivatives for the entire training set, once an iteration is over you can throw away the derivatives of that mini-batch.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.