What is the intuition of momentum term in the neural network back propagation?

Question

I am going through the derivation of neural network using this lecture pdf

And I am stuck on equation $(21)$

Note on notation:

1. Activation function of layer $j$ is $y_j$
2. Summation of weights of layer $j$ is $x_j$
3. final label is $t$

I am trying to figure out where

$$\eta \Delta w_{kj} (n-1)$$

is coming from in the final equation $(21)$

$$ \Delta w_{kj}(n) = \alpha \delta_j y_k + \eta \Delta w_{kj} (n-1)$$

The author mentioned that it is a momentum term without really elaborating on it.

I thought $\Delta w_{kj}$ calculation is the following

$$\Delta w_{kj} = - \alpha \frac{\partial E}{\partial w_{kj}}$$

for 1 layer before final output layer:

$$\Delta w_{kj} = - \alpha (-(t_j-y_j))y_j(1-y_j)y_k$$

for all other layers:

$$\Delta w_{kj} = - \alpha (\delta_{i}w_{ji}) y_j(1-y_j)y_k$$

So what is the momentum term?

Can someone help me out ?

Answer 1

Since you are asking for intuition, here's the general idea: Imagine that you are using stochastic gradient descent (SGD) to traverse the surface of the loss function. Further imagine that that surface looks like a mountain and that you are starting near the top. The surface has many small crags and lips but is clearly headed down to a valley. As you traverse the surface using SGD, the magnitude - and sign - of the derivative changes often, and so there is a great deal of fluctuation in the direction and speed that you take down the mountain. If you include a momentum term, then instead of just relying on the gradient at each new stopping point to determine your movement, your direction now also depends on the size and direction of your movement in the previous update. And since the amount and direction you moved last update also depended on the update before it you can see that each movement depends on the entire history of previous movements. So pretty soon you're running down the mountain toward the valley.

Notice that the further you are in time from an update, the less important it is for determining your current update, since, if you expand the momentum term at time $t$ then the contribution of the update from time $t-k$ is: \begin{equation} \alpha^{k+1}\Delta\omega_{t-k} \end{equation} Where $\alpha\in[0,1]$ is the momentum and $\Delta\omega_{t-k}$ is the weight update at time $t-k$

Answer 2

Firstly, let's pretend we have just started training so that: \begin{equation} n = 0 \end{equation} and \begin{equation} \omega_{kj}(0) = \alpha\delta_jy_k \end{equation} because this is the first time step. If we go one training step further to \begin{equation} n = 1 \end{equation} then: \begin{equation} \omega_{kj}(1) = \alpha\delta_jy_k + \eta\omega_{kj}(0) \end{equation} and one more to \begin{equation} n = 2 \end{equation} \begin{equation} \omega_{kj}(2) = \alpha\delta_jy_k + \eta\omega_{kj}(1) \end{equation} and substitute: \begin{equation} \omega_{kj}(2) = \alpha\delta_{j2}y_{k2} + \eta(\alpha\delta_{j1}y_{k1} + \eta\omega_{kj}(0)) \end{equation}

You can see that if we continued this trend, even a time step of

\begin{equation} n = 200 \end{equation} would still slightly be influenced by the very first weight gradient at time step 0 (but very very slightly). This is the concept of momentum: velocity with a memory of past velocities. This concept is directly related to physics.

If I push a block 10m/s forward (think of that as my first time step) and then I push it at -20m/s (my second time step); then via momentum this would be:

new speed = -20m/s + u(100m/s) where u is friction (or in our case eta). So if friction is 0.1, my final speed will actually be -20 + (0.1)(100) = -10. I am closer to -20 but I am not completely there. If I push the block again at -20m/s my result is: -10 + (0.1)(-20) = -12, and I am closer to the speed I wish to attain.

In the concept of neural networks, this means if I am going down the gradient and searching for the minimum, I do not want to drastically change directions at each time step because some directions could lead to valleys or local minima. Therefore I want to go the direction in which most of my weight gradients push me. Changing this direction usually requires a large number of weight gradients against my current gradient 'flow'.