gradient descent momentum vs step size In the gradient descent method, the learning rate (which is multiplied by the results of the gradient on each weight) identifies the size of the step (steep down) that the algorithm takes in each iteration to reduce the objective cost function. How does it differ from the parameter momentum?
Thank you
 A: Momentum is a whole different method, that uses parameter that works as an average of previous gradients.
Precisely in Gradient Descent (let's denote learning rate by $\eta$)
$$w_{i+1} = w_i - \eta \nabla F(w)$$
Whereas in Momentum Method
$$w_{i+1} = w_i - \gamma v_i$$
Where
$$v_{i+1} = \beta v_i + (1 - \beta)  \nabla F(w)$$
Note that this method has two hyperparameters, instead of one like in GD, so I can't be sure if your momentum means $\gamma$ or $\beta$. If you use some software though, it should have two parameters.
A: Classical gradient descent has the shape:
$$\mathbf{x}_{i+1} = \mathbf{x}_i - \gamma \cdot\nabla f(\mathbf{x}_i)$$
for a differentiable function $f$, some $\gamma \in \mathbb{R}$ (the "learning rate" you referred to) and a starting point $\mathbf{x}_{0}$.
With "momentum" is meant the following adaption,
$$\begin{align}
\mathbf{x}_{i+1} &= \mathbf{x}_i - \alpha \cdot \mathbf{z}_{i+1} \\
\text{whereat}\quad \mathbf{z}_{i+1} &= \beta \cdot \mathbf{z}_{i} + \nabla f(\mathbf{x}_i)
\end{align}$$
where $\mathbf{z}$ can be interpreted as a momentum.
If you write down the closed form, you get
$$\mathbf{x}_{i+1} = \mathbf{x}_{0} - \gamma \cdot \sum^i_{k=0} \nabla f(\mathbf{x}_k)$$
for the "classical" version and
$$\mathbf{x}_{i+1} = \mathbf{x}_{0} - \alpha \cdot \sum^i_{k=0} \dfrac{1-\beta^{i-k+1}}{1-\beta}\nabla f(\mathbf{x}_k)$$
for the "momentum" version.
As you can see, they are both pretty much the same (the class is called Linear First Order Methods). Momentum just puts greater emphasis on the recent gradients, whereas the classic version treats them equally (as illustrated below).

