4
$\begingroup$

I'm attempting the final problem in Chapter 2 of Michael Nielsen's Neural Networks and Deep Learning book. The network in question is a simple 3-layer feedforward neural network with sigmoid activation functions for the neurons in the hidden and output layers.

When calculating the gradient, it appears I need to manage derivatives of the cost function with respect to each weight and bias parameter separately for each training example, for each stage of backpropagation until the final step; this leads to some rather ugly code. Is there any way to get around this?

Improve this question
$\endgroup$
3
  • $\begingroup$ I asked a similar question, and somebody answered basically "yes, you're stuck". I'm not entirely confident in the answer however... stats.stackexchange.com/questions/240971/… $\endgroup$
    – generic_user
    Dec 12, 2016 at 19:19
  • $\begingroup$ There is a trick for turning the bias into a feature and then you don't have to treat it differently than any of the weights. Is that the type of thing you're looking for? $\endgroup$
    – Aaron
    Dec 12, 2016 at 23:32
  • $\begingroup$ No, the issue is that I need to contend with a 3D array of weights for each connected pair of layers in the neural network, and it seems I can't sum out the third dimension in these arrays until the very end of the backpropagation process. The same problem occurs whether or not I treat the bias term as a "weight". $\endgroup$
    – jsaporta
    Dec 13, 2016 at 0:33

1 Answer 1

Reset to default
1
$\begingroup$

Within an iteration, you usually have to store the derivatives because you'll use them to update the weights after backpropagation. Especially some optimization algorithms (e.g. momentum, ADAM) also need the derivatives to maintain some internal status.

If at some point you can be sure that some derivatives won't be involved in the rest of the iteration you can erase them immediately to save some memory, though I don't think this will make the code simpler.

You don't have to keep the derivatives for the entire training set, once an iteration is over you can throw away the derivatives of that mini-batch.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.