Can all neural network cost functions be written as an average of individual cost and as a function of the activations at the output?

Question

In chapter 2 of Michael Nielsen's Neural Networks and Deep Learning it says backpropagation relies on

• The first assumption we need is that the cost function can be written as an average $C = \frac{1}{n} \sum_{x} C_x$ over cost functions $C_x$ for individual training examples, $x$.

  I can accept this is because propagation only allows us to compute $\frac{\partial C_x}{\partial w}$ and $\frac{\partial C_x}{\partial b}$ for a single training example $x$. But are all cost functions in the literature really an average of cost for individual training examples? If a cost function is not defined this way, is backpropagation completely useless?

• The second assumption we make about the cost is that it can be written as a function of the outputs from the neural network

  How else would you define a cost function? At some level doesn't it have to be a function of the outputs from the neural network? How else would you measure the error between a training sample's label and what your neural network is doing?

Answer 1

I believe both these assumptions are not always necessary and often broken.

Batch normalization normalizes values in hidden layers / feature maps across all inputs in a mini-batch when doing mini-batch SGD. This means the cost function can no longer be split up into individual costs for each input. Batch norm is a pretty ubiquitous component of modern neural networks.

The second claim depends on what you would consider an "output" of the network. Weight decay / regularization adds on to the cost function the L2 norm of the weights, which I think can hardly be called an output.

The only real requirement for backpropagation is that you can define gradients on all the computation steps between weights you want to backprop into, and the cost function you want to backprop from. These gradients don't necessarily need to be mathematically well defined, or even correct and unbiased (see straight-through gradient estimators).