# Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me.

In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial a_j}\dfrac{\partial a_j}{\partial w_{ji}}$ (equation (5.50)), a notation is defined as $\delta_j\equiv\dfrac{\partial E_n}{\partial a_j}$ (equation (5.51)). In my understanding $\delta_j$ is the first part in the chain rule. However in equation (5.54), the book mentioned

As we have seen already, for the output units, we have $$\delta_k=y_k-t_k$$

Question 1: $y_k-t_k$ is the error on output unit $k$, which is simply the difference between the $k$th output unit value and the corresponding target value. But, from the definition of the notation $\delta_k$, we should have $$\delta_k=\dfrac{\partial \frac{1}{2}(y_k-t_k)^2}{\partial a_k}=(y_k-t_k)\dfrac{\partial y_k}{\partial a_k}=(y_k-t_k)\dfrac{\partial h(a_k)}{\partial a_k}$$ where $h(a_k)$ is the activation function. So why in the book $\delta_k=y_k-t_k$??

In page 242, Section 5.3. Error Backpropagation,

Consider a simple linear model where the outputs $y_k$ are linear combinations of the input variables $x_i$ so that $y_k=\sum_iw_{ki}x_i$. For a particular input pattern $n$, the error function is $E_n=\dfrac{1}{2}\sum_k(y_{nk}-t_{nk})^2$, where $y_{nk}=y_k(\boldsymbol{x_n},\boldsymbol{w})$. So the gradient of this error function with respect to a weight $w_{ij}$ is given by $$\frac{\partial E_n}{\partial w_{ji}}=(y_{nj}-t_{nj})x_{ni}$$ which can be interpreted as a ‘local’ computation involving the product of an ‘error signal’ $y_{nj} − t_{nj}$ associated with the output end of the link $w_{ji}$ and the variable $x_{ni}$ associated with the input end of the link.

Question 2: I am not clear with the structure of this neural network. The one in the book is a two-layer neural network with linear activation, is it?

Q1: It seems here $y_k=a_k$, in PRML around eq 5.18
In the regression case, we can view the network as having an output activation function that is the identity, so that $y_k = a_k$
Consider first a simple linear model in which the outputs $y_k$ are linear combinations of the input variables $x_i$
• Q1: Isn't the activation function in the context is given in eq. 5.49, i.e., $z_j=h(a_j)$? Also in another book I'm reading Python Machine Learning - Second Edition, for a 3-layer MLP with sigmoid activation function on hidden and output units, the error vector of the output layer has a similar form: $\boldsymbol{\delta^{(out)}=a^{(out)}-y}$, where $a^{(out)}$ is the output vector and $y$ is the target vector. – Tyler 十三将士归玉门 Jul 31 '18 at 8:39
• @Tyler提督九门步军巡捕五营统领 you're right, in the first quote it mentioned the activation function used is the identity function since it's a regression task, so $h(a_j)=a_j$ :) – dontloo Jul 31 '18 at 9:05