The derivation of $\delta_j = \frac{\partial E_n}{ \partial a_j}$ errors for hidden units in back propagation for neural networks with the chain rule

Question

I was trying to understand the derivation for back propagation for multi-layer neural networks from Bishop's Pattern Recognition and Machine Learning book. Specifically I was reading section 5.3.1 from page 242 to 244.

The equation that is specifically confusing me is equation 5.55

$$ \delta_j \equiv \frac{\partial E_n}{ \partial a_j} = \sum_k \frac{\partial E_n}{ \partial a_k} \frac{\partial a_k}{ \partial a_j} $$

where Bishop goes on to say:

where the sum runs over all units $k$ to which $j$ sends connections.

furthermore, what confuses me is the use of the chain rule and how the connections the affect the partial derivative.

To make this discussion easier recall the definition of $E_n$ (the error of a the neural network for the nth training point, equation 5.46):

$$E_n = \frac{1}{2} \sum_k (y_{nk} - t_{nk} )^2 = \frac{1}{2} \|y_n - t_n \|^2$$

where $y_n $ is the vector of outputs of our neural and $t_n$ is the true target output we are trying to learn.

My confusion is specifically how he applied the multivariable chain rule. Usually the way I think the multivariable chain rule is as follows; given a function $f(x_1(t), ..., x_N(t) )$, if we want its derivative wrt to t then we get:

$$\frac{df}{dt} = \sum^N_{k=1} \frac{\partial f}{\partial x_k} \frac{d x_k}{d t}$$

However, I am having some difficulties understanding how that equations was applied in the context of deriving $\frac{\partial E_n}{ \partial a_j}$ for the hidden units of the neural network.

In particular, I thought $E_n$ is a function of all the $a_j$'s, at every layer, so why wouldn't all the $a_j$'s from each layer be part of the summation but only the ones from one layer before?

Also, the part $ \frac{\partial a_k}{ \partial a_j} $ from equation 5.55 is never zero, right? because the $a_j$ vs $a_k$ are from different nodes in the network, right? Wouldn't it be easier/better to indicate this dependence with superscripts indicating the layers?

Is somebody knows how to explain how that equation come about, I would be super grateful!

---

For reference I will paste the relevant section of the chapter as reference:

Answer 1

I will include superscripts and bias term to make it more clear. It seems to me $ \frac{\partial a_k}{a_j}$ is totally redundant as it is equal to 1 when $ k=j$ and 0 $ otherwise$. It would be better to use notation:

$ \delta_j^l = \frac{\partial E}{a_j^l} = \sum_k \frac{\partial E}{z_k^l} \frac{\partial z_k^l}{a_j^{l-1}}$ (Eq.1)

where $ z_k^l$ is activation of k-th unit at l-th layer, $ a_j^l$ is pre-activation of k-th unit of l-th layer.

Here is a few equation before we process;

$ a_k^l = \sum_k W_{kj}^{l}*z_j^{l} + b_k^{l}$ (Eq.2)

$ z_k^{l+1} = h(a_k^l)$ (Eq.3) where $ h(.)$ is activation function and next layer starts after activation function.

$ \frac{\partial z_k^{l+1}}{\partial a_k^l} = h'(a_k^l)$ (Eq.4)

$ \frac{\partial a_k^l}{\partial W_{kj}^{l}} = z_j^{l}$ (Eq.5)

$ \frac{\partial a_k^l}{\partial b_k^{l}} = 1$ (Eq.6)

$ E = \frac{1}{2} \sum_k(z_k^L−t_k)^2$, (Eq.7) where $ L$ is the index of last layer;

$ \delta_j^L = \frac{\partial E}{a_j^{L-1}} = \sum_k \frac{\partial E}{z_k^L} \frac{\partial z_k^L}{a_j^{L-1}} = (z_k^L−t) .* h'(a_k^l)$, (Eq.8) in here I'm assuming activation of each unit only depends on its own pre-activation. So, if softmax is used it changes a bit. As we have error term at the last layer, gradients with respect to weights and biases at the previous layer can be evaluated by:

$ \frac{\partial E}{\partial W_{kj}^{L-1}} = \frac{\partial E}{\partial a_k^{L-1}} \frac{\partial a_k^{L-1}}{\partial W_{kj}^{L-1}} = \delta_k^L \frac{\partial a_k^{L-1}}{\partial W_{kj}^{L-1}} = \delta_j^L z_k^{L-1}$, from Eq.5

$ \frac{\partial E}{\partial b_{k}^{L-1}} = \frac{\partial E}{\partial a_k^{L-1}} \frac{\partial a_k^{L-1}}{\partial b_{k}^{L-1}} = \delta_k^L$, from Eq.6

In order to evaluate gradients of previous layer, error at the current layer s evaluated by using error at the later layer.

$ \delta_j^{L-1} = \frac{\partial E}{\partial a_{j}^{L-2}} = \delta_k^{L} \frac{\partial a_j^{L-1}}{\partial z_k^{L-1}} \frac{\partial z_j^{L-1}}{\partial a_k^{L-2}} = (\delta_k^{L}* W_{kj}^{L-1}).*h'(a_j^{L-2}) $

To sum up, summation over all units on a layer is only used when activation of a units depends on pre-activation of other units in the same layer like softmax. So, if units are independent you can more easly apply chain rule as in:

$ \frac{\partial E}{\partial a_{N_1}^{1}} = \frac{\partial E}{\partial a_{N_L}^{L}} \frac{\partial a_{N_L}^{L}}{\partial a_{N_{L-1}}^{L-1}} \frac{\partial a_{N_{L-1}}^{L-1}}{\partial a_{N_{L-2}}^{L-2}}.... \frac{\partial a_{N_3}^{3}}{\partial a_{N_2}^{2}} \frac{\partial a_{N_2}^{2}}{\partial a_{N_1}^{1}}$ where $ N_l $ is number of units at layer $ l$