# What is the maximum size of weights update in Momentum optimisation?

Given the optimization rules of momemntum gradient descent for weights update of a neural network:

$$m\leftarrow \beta m -\eta \nabla_\theta J(\theta)$$ $$\theta \leftarrow \theta + m$$

where $$\theta$$ are the weights, $$J$$ the cost functionthe book Hands on machine learning says

You can easily verify that if the gradient remains constant, the terminal velocity (i.e., the maximum size of the weight updates) is equal to that gradient multiplied by the learning rate η multiplied by $$\frac{1}{1-\beta}$$ (ignoring the sign). For example, if $$β = 0.9$$, then the terminal velocity is equal to 10 times the gradient times the learning rate, so Momentum optimisation ends up going 10 times faster than Gradient Descent

Now I get that the the maximum size update is given by the max value of $$m$$ and it seems to me it does something like $$m (1-\beta) = -\eta K$$ where $$K=\nabla_\theta J(\theta)$$ is the constant gradient, but left-side $$m$$ and right-side $$m$$ are at different instants so they are basically different.

Can someone explain the statement?

$$m_1 = -\eta K$$
$$m_2 = -\beta\eta K-\eta K$$
...
$$m_n = - \beta^{n-1}\eta K - \beta^{n-2} \eta K - ... -\eta K$$
$$m_n = -\eta K (\beta^0 + \beta^1 + ..\beta^{n-1})$$

Since $$0< \beta < 1$$, according to power series / geometric series (https://en.wikipedia.org/wiki/Power_series),

$$\beta^0 + \beta^1 + ...+\beta^{n-1} = \frac{1}{1-\beta}$$

Therefore,

$$m_n = -\frac{\eta K}{1-\beta}$$

• Just to add, that the last formula holds when $n \to \infty$. Commented Jun 8, 2023 at 17:30