How the equation 5.80 in _Pattern Recognition and Machine Learning_ by Bishop is derived?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I tried to figure out the maths behind it and this is what I understood. The perceptron architecture having the two layers ($j$,$k$) is shown below. The activation function at node $j$ is $h(.)$ and $k$ being the output node has unit activation.
We have to find $\frac{\partial E_n}{\partial a_j^2}$. This can be evaluated as $\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_j}\bigg)$. Referring the perceptron diagram and using chain rule to back propagate the error across the path from $y_k$ to $a_j$, $\frac{\partial E_n}{\partial a_j}$ can be evaluated by first differentiating $E_n$ w.r.t. $a_k$, then $a_k$ w.r.t. $z_j$ and lastly $z_j$ w.r.t. $a_j$. At node $k$, we have to consider all the incoming connections and hence have to sum over all the possible values of $k$. The expression for $\frac{\partial E_n}{\partial a_j}$ is
$$\begin{align} \frac{\partial E_n}{\partial a_j} = \sum_{k} \frac{\partial E_n}{\partial a_k} \frac{\partial a_k}{\partial z_j} \frac{\partial z_l}{\partial a_j} = \sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj} h^{'}(a_j) = h^{'}(a_j)\sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj} \tag{A} \end{align}$$
Now, differentiating $\frac{\partial E_n}{\partial a_j}$ w.r.t. $a_j$ (using product rule of differentiation), we have
$$\begin{align} \frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_j}\bigg) = \frac{\partial}{\partial a_j} \bigg(h^{'}(a_j)\sum_{k} \frac{\partial E_n}{\partial a_k} w_{kj}\bigg) = \frac{\partial}{\partial a_j} \bigg(h^{'}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k}\bigg) \end{align}$$
$$\begin{align} = h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j)\frac{\partial}{\partial a_j} \bigg(\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k}\bigg) \end{align}$$
$$\begin{align} = h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j) \sum_{k} w_{kj}\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg) \tag{B} \end{align}$$
The differential $\frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg)$ can be evaluated using $(A)$ as
$$\begin{align} \frac{\partial}{\partial a_j}\bigg(\frac{\partial E_n}{\partial a_k}\bigg) = h^{'}(a_j)\sum_{k^{'}} \frac{\partial^2 E_n}{\partial a_k^{'}a_k} w_{k^{'}j} \tag{C} \end{align}$$
Substituting $(C)$ in $(B)$, the final expression is
$$\begin{align} \frac{\partial^2 E_n}{\partial a_j^{2}} = h^{''}(a_j)\sum_{k} w_{kj}\frac{\partial E_n}{\partial a_k} + h^{'}(a_j)^2 \sum_{k}\sum_{k^{'}} w_{kj}w_{k^{'}j}\frac{\partial^2 E_n}{\partial a_k^{'}a_k} \end{align}$$