1
$\begingroup$

I'm trying to understand the description of the error backpropagation algorithm as explained in Christopher Bishop's book, in particular, section 5.3.1 "Evaluation of error-function derivative".

The thing that really bothers me is Eq. (5.54) for the derivative of the cost function with respect to the weights the activations of the output layer:

$\delta_k(:= \frac{\text{d}E_n}{\text{d}a_k}) = y_k - t_k$

Is it implicitly based on the assumption of linearity for the output units? It's not mentioned anywhere explicitly, and I cannot explain the absence of the derivative of the activation function of the output neurons in Eq. (5.54).

$\endgroup$
1
$\begingroup$

Equation (5.54) does not represent the derivative of the cost function with respect to weights to the output layer. If you take a look, Equation (5.47) gives you the derivative. Eq (5.54) is the derivative of cost function with respect to $y_i$. You need to compute it in order to get the derivative w.r. t. $w_{i,j}$. Think of it as the accumulated derivative that you computed in backwards. Consider the following.

$$Cost = \frac 1 2 \sum_{k=1}^{m}(y_k-t_k)^2$$ $$\frac{\partial Cost}{\partial y_i} = y_i - t_i $$ $$\frac{\partial y_i}{w_{i,j}} = \frac{\partial(w_{k,0}+\sum_{l=1}^{p}w_{i,l}z_l)}{\partial w_{i,j}} = z_j$$

So:

$$\frac{\partial Cost}{\partial w_{i,j}} = \frac{\partial Cost}{\partial y_i}\frac{\partial y_i}{\partial w_{i,j}} = (y_i-t_i)z_j$$

When you traverse the network in backwards, you first compute $y_i-t_i$ when you reach the output layer, and when you get to the weights between the output layer and its predecessor, you compute the derivative w.r.t. the weight in question. Actually, this is simply an application of the chaining rule in calculus.

If you continue to move backwards, you will realize that the derivatives of the weights between the layers $r$ and $r+1$ can be determined by multiplying the accumulated derivative (from the end up to $r+1$) with the derivative of the activations of $r+1$ w.r.t the weights. This is what $\delta$s are for.

For more information, read this chapter. It helped me understand backpropagation.

Maybe my notation is not the same as the one you use, but it should not be hard to convert to the one you are familiar with.

I hope I helped. :)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a lot! You helped a lot! You're completely right with respect to what $E_n$ is differentiated with respect to. If you however involve the definition Eq. (5.51) of $\delta_j$, still you have to differentiate $E_n$ with respect to $a_j$ to obtain $\delta_j$ and that should produce a derivative of the nonlinearity of the output unit, shouldn't it? $\endgroup$ – Grisha Jul 14 '16 at 7:58
  • $\begingroup$ Since $E_i$s are quadratic functions of $a_j$s (for the output layers, $a_i=y_i$), the output values (the final activations) figure in linear way in $E_i$s' derivatives. Those are the first $\delta$s. They show you how the outputs should be changed in order to minimize the error. Inferring the needed changes of the weights based on this information involves further derivation with respect to the weights that are arguments to $y_i$s. :) $\endgroup$ – Milos Jul 15 '16 at 21:05
  • $\begingroup$ In brief, when you compute $\delta_j$ for the output layer, you treat $y_i$ as a bare input. Later, for the weights that lead impulses from the second last to the last layer, you treat $y_i$ as $sigmoid(sum(weights))$, which is why you include $sigmoid'(sum(weights))*sum'(weights)$. P.S. $sum(weights)$ is a linear sum of weights multiplied by appropriate outputs from the previous layer, and $'$ is appropriate derivative. I used this informal, yet shorthand notation for the purpose of brevity. $\endgroup$ – Milos Jul 15 '16 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.