# Suspicion about the errorback propagation formula in PRML

From PRML: Do variations in $a_j$ give rise to variations in $E$ only via $a_k$? True. Is $h'(a_j)$ the same across all $k$? I'm afraid not.

Let's write down what $\partial a_k/\partial a_j$ are. Since $a_k$ are in the layer that succeeds $a_j$:

$$a_k = h(\sum_{j'} w_{j'\to k}a_{j'})$$

in which $j'$ runs through all nodes in the layer of $a_j$. As such

$$\dfrac{\partial a_k}{\partial a_j} = h'(\sum_{j'} w_{j'\to k}a_j)w_{j\to k}.$$

And thus I think the correct error backpropagation formula should be

$$\dfrac{\partial E}{\partial a_j}=\sum_k h'(\sum_{j'} w_{j'\to k}a_{j'})w_{j\to k}\cdot\dfrac{\partial E}{\partial a_k}.$$

And clearly, the $h'$ cannot be factored out of the summation operator, because for different $k$ it is evaluated at different points i.e. $\sum_{j'} w_{j'\to k}a_{j'}$ clearly dependent on $k$. (And in fact they don't depend on $j$, which the notation in PRML incorrectly indicates.)

Thoughts?

PS: In PRML, one layer of neurals is defined as a linear map composed by a (elementwise) nonlinear activation function when we treat one layer of nodes as entries of a vector.

It seems a very common misconception about the definition of activation ($a_j$) in neural networks.
It is defined in formula 5.48 and 5.49 that $a_j = \sum_i w_{ji}z_i$ and $z_j=h(a_j)$, so it follows $a_j = \sum_i w_{ji}h(a_i)$ instead of the first formula in the post. Then $$\frac{\partial a_k}{\partial a_j}=w_{kj}h'(a_j)$$ $$\frac{\partial E}{\partial a_j}=\sum_k\frac{E}{\partial a_k}\frac{\partial a_k}{\partial a_j}=h'(a_j)\sum w_{kj}\delta_k$$
• Never mind. I got it. So $a_j$ generally denote the "pre-activated" linear inputs to the $j$-th layer instead of the "post-activated" values. – Vim Jun 1 '17 at 10:55