I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me.

In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial a_j}\dfrac{\partial a_j}{\partial w_{ji}}$ (equation (5.50)), a notation is defined as $\delta_j\equiv\dfrac{\partial E_n}{\partial a_j}$ (equation (5.51)). In my understanding $\delta_j$ is the first part in the chain rule. However in equation (5.54), the book mentioned

As we have seen already, for the output units, we have $$\delta_k=y_k-t_k$$

Question 1: $y_k-t_k$ is the error on output unit $k$, which is simply the difference between the $k$th output unit value and the corresponding target value. But, from the definition of the notation $\delta_k$, we should have $$\delta_k=\dfrac{\partial \frac{1}{2}(y_k-t_k)^2}{\partial a_k}=(y_k-t_k)\dfrac{\partial y_k}{\partial a_k}=(y_k-t_k)\dfrac{\partial h(a_k)}{\partial a_k}$$ where $h(a_k)$ is the activation function. So why in the book $\delta_k=y_k-t_k$??

In page 242, Section 5.3. Error Backpropagation,

Consider a simple linear model where the outputs $y_k$ are linear combinations of the input variables $x_i$ so that $y_k=\sum_iw_{ki}x_i$. For a particular input pattern $n$, the error function is $E_n=\dfrac{1}{2}\sum_k(y_{nk}-t_{nk})^2$, where $y_{nk}=y_k(\boldsymbol{x_n},\boldsymbol{w})$. So the gradient of this error function with respect to a weight $w_{ij}$ is given by $$\frac{\partial E_n}{\partial w_{ji}}=(y_{nj}-t_{nj})x_{ni}$$ which can be interpreted as a ‘local’ computation involving the product of an ‘error signal’ $y_{nj} − t_{nj}$ associated with the output end of the link $w_{ji}$ and the variable $x_{ni}$ associated with the input end of the link.

Question 2: I am not clear with the structure of this neural network. The one in the book is a two-layer neural network with linear activation, is it?


1 Answer 1


Q1: It seems here $y_k=a_k$, in PRML around eq 5.18

In the regression case, we can view the network as having an output activation function that is the identity, so that $y_k = a_k$

Q2: In the discussion on P242 there's only one layer with no activation function, actually the author didn't call it a neural network but a "simple linear model"

Consider first a simple linear model in which the outputs $y_k$ are linear combinations of the input variables $x_i$

  • $\begingroup$ Q1: Isn't the activation function in the context is given in eq. 5.49, i.e., $z_j=h(a_j)$? Also in another book I'm reading Python Machine Learning - Second Edition, for a 3-layer MLP with sigmoid activation function on hidden and output units, the error vector of the output layer has a similar form: $\boldsymbol{\delta^{(out)}=a^{(out)}-y}$, where $a^{(out)}$ is the output vector and $y$ is the target vector. $\endgroup$ Jul 31, 2018 at 8:39
  • $\begingroup$ @Tyler提督九门步军巡捕五营统领 you're right, in the first quote it mentioned the activation function used is the identity function since it's a regression task, so $h(a_j)=a_j$ :) $\endgroup$
    – dontloo
    Jul 31, 2018 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.