In chapter 2 of Michael Nielsen's Neural Networks and Deep Learning it says backpropagation relies on

  • The first assumption we need is that the cost function can be written as an average $C = \frac{1}{n} \sum_{x} C_x$ over cost functions $C_x$ for individual training examples, $x$.

    I can accept this is because propagation only allows us to compute $\frac{\partial C_x}{\partial w}$ and $\frac{\partial C_x}{\partial b}$ for a single training example $x$. But are all cost functions in the literature really an average of cost for individual training examples? If a cost function is not defined this way, is backpropagation completely useless?

  • The second assumption we make about the cost is that it can be written as a function of the outputs from the neural network

    How else would you define a cost function? At some level doesn't it have to be a function of the outputs from the neural network? How else would you measure the error between a training sample's label and what your neural network is doing?


1 Answer 1


I believe both these assumptions are not always necessary and often broken.

Batch normalization normalizes values in hidden layers / feature maps across all inputs in a mini-batch when doing mini-batch SGD. This means the cost function can no longer be split up into individual costs for each input. Batch norm is a pretty ubiquitous component of modern neural networks.

The second claim depends on what you would consider an "output" of the network. Weight decay / regularization adds on to the cost function the L2 norm of the weights, which I think can hardly be called an output.

The only real requirement for backpropagation is that you can define gradients on all the computation steps between weights you want to backprop into, and the cost function you want to backprop from. These gradients don't necessarily need to be mathematically well defined, or even correct and unbiased (see straight-through gradient estimators).

  • $\begingroup$ Good answer, +1. Another nit-pick about the "output of the network", you need the true target to define the cost as well, and that's certainly not an output of the network! $\endgroup$
    – kbrose
    Commented Sep 19, 2018 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.