I am going through the derivation of neural network using this lecture pdf
And I am stuck on equation $(21)$
Note on notation:
- Activation function of layer $j$ is $y_j$
- Summation of weights of layer $j$ is $x_j$
- final label is $t$
I am trying to figure out where
$$\eta \Delta w_{kj} (n-1)$$
is coming from in the final equation $(21)$
$$ \Delta w_{kj}(n) = \alpha \delta_j y_k + \eta \Delta w_{kj} (n-1)$$
The author mentioned that it is a momentum term without really elaborating on it.
I thought $\Delta w_{kj}$ calculation is the following
$$\Delta w_{kj} = - \alpha \frac{\partial E}{\partial w_{kj}}$$
for 1 layer before final output layer:
$$\Delta w_{kj} = - \alpha (-(t_j-y_j))y_j(1-y_j)y_k$$
for all other layers:
$$\Delta w_{kj} = - \alpha (\delta_{i}w_{ji}) y_j(1-y_j)y_k$$
So what is the momentum term?
Can someone help me out ?