1
$\begingroup$

From PRML:

enter image description here

Do variations in $a_j$ give rise to variations in $E$ only via $a_k$? True. Is $h'(a_j)$ the same across all $k$? I'm afraid not.

Let's write down what $\partial a_k/\partial a_j$ are. Since $a_k$ are in the layer that succeeds $a_j$:

$$a_k = h(\sum_{j'} w_{j'\to k}a_{j'})$$

in which $j'$ runs through all nodes in the layer of $a_j$. As such

$$\dfrac{\partial a_k}{\partial a_j} = h'(\sum_{j'} w_{j'\to k}a_j)w_{j\to k}.$$

And thus I think the correct error backpropagation formula should be

$$\dfrac{\partial E}{\partial a_j}=\sum_k h'(\sum_{j'} w_{j'\to k}a_{j'})w_{j\to k}\cdot\dfrac{\partial E}{\partial a_k}.$$

And clearly, the $h'$ cannot be factored out of the summation operator, because for different $k$ it is evaluated at different points i.e. $\sum_{j'} w_{j'\to k}a_{j'}$ clearly dependent on $k$. (And in fact they don't depend on $j$, which the notation in PRML incorrectly indicates.)

Thoughts?

PS: In PRML, one layer of neurals is defined as a linear map composed by a (elementwise) nonlinear activation function when we treat one layer of nodes as entries of a vector.

$\endgroup$
1
$\begingroup$

It seems a very common misconception about the definition of activation ($a_j$) in neural networks.

It is defined in formula 5.48 and 5.49 that $a_j = \sum_i w_{ji}z_i$ and $z_j=h(a_j)$, so it follows $a_j = \sum_i w_{ji}h(a_i)$ instead of the first formula in the post. Then $$\frac{\partial a_k}{\partial a_j}=w_{kj}h'(a_j)$$ $$\frac{\partial E}{\partial a_j}=\sum_k\frac{E}{\partial a_k}\frac{\partial a_k}{\partial a_j}=h'(a_j)\sum w_{kj}\delta_k$$

$\endgroup$
  • 1
    $\begingroup$ Never mind. I got it. So $a_j$ generally denote the "pre-activated" linear inputs to the $j$-th layer instead of the "post-activated" values. $\endgroup$ – Vim Jun 1 '17 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.