Error Backpropagation, Christopher Bishop “Pattern Recognition and Machine Learning” I'm trying to understand the description of the error backpropagation algorithm as explained in Christopher Bishop's book, in particular, section 5.3.1 "Evaluation of error-function derivative".
The thing that really bothers me is Eq. (5.54) for the derivative of the cost function with respect to the weights the activations of the output layer:
$\delta_k(:= \frac{\text{d}E_n}{\text{d}a_k}) = y_k - t_k$
Is it implicitly based on the assumption of linearity for the output units? It's not mentioned anywhere explicitly, and I cannot explain the absence of the derivative of the activation function of the output neurons in Eq. (5.54).
 A: Equation (5.54) does not represent the derivative of the cost function with respect to weights to the output layer. If you take a look, Equation (5.47) gives you the derivative. Eq (5.54) is the derivative of cost function with respect to $y_i$. You need to compute it in order to get the derivative w.r. t. $w_{i,j}$. Think of it as the accumulated derivative that you computed in backwards. Consider the following.
$$Cost = \frac 1 2 \sum_{k=1}^{m}(y_k-t_k)^2$$
$$\frac{\partial Cost}{\partial y_i} = y_i - t_i $$
$$\frac{\partial y_i}{w_{i,j}} = \frac{\partial(w_{k,0}+\sum_{l=1}^{p}w_{i,l}z_l)}{\partial w_{i,j}} = z_j$$
So:
$$\frac{\partial Cost}{\partial w_{i,j}} = \frac{\partial Cost}{\partial y_i}\frac{\partial y_i}{\partial w_{i,j}} = (y_i-t_i)z_j$$
When you traverse the network in backwards, you first compute $y_i-t_i$ when you reach the output layer, and when you get to the weights between the output layer and its predecessor, you compute the derivative w.r.t. the weight in question. Actually, this is simply an application of the chaining rule in calculus.
If you continue to move backwards, you will realize that the derivatives of the weights between the layers $r$ and $r+1$ can be determined by multiplying the accumulated derivative (from the end up to $r+1$) with the derivative of the activations of $r+1$ w.r.t the weights. This is what $\delta$s are for.
For more information, read this chapter. It helped me understand backpropagation.
Maybe my notation is not the same as the one you use, but it should not be hard to convert to the one you are familiar with.
I hope I helped. :)
