Here's a list of popular algorithms used in training neural networks:

• SGD updates parameters in the negative direction of the gradient, learning rate parameter $\epsilon_k$ should be decreased over time, computation time is proportional to mini-batch size $m$: $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ \theta = \theta - \epsilon_k \times g $$

• Momentum accumulates exponentially decaying moving average of past gradients and continues to move in their direction, thus step size depends on how large and how aligned the sequence of gradients are, common values of momentum parameter $\alpha$ are 0.5 and 0.9 $$ v = \alpha v - \epsilon \nabla_\theta \big(\frac{1}{m}\sum_i L(f(x^{(i)};\theta), y^{(i)}) \big) \\ \theta = \theta + v $$

• Nesterov Momentum inspired by Nesterov's accelerated gradient method, difference betweent Nesterov and standard momentum is where the gradient is evaluated, with Nesterov's momentum the gradient is evaluated after the current velocity is applied, thus Nesterov's momentum adds a correction factor to the gradient: $$ v = \alpha v - \epsilon \nabla_\theta \big(\frac{1}{m}\sum_i L(f(x^{(i)};\theta + \alpha \times v), y^{(i)}) \big) \\ \theta = \theta + v $$

• AdaGrad adapts the learning rates of all model parameters, learning rate is inversely proportional to the square root of the sum of historical squared values, weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased: the net effect is greater progress in the more gently sloped directions of parameter space: $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ s = s + g^{T}g \\ \theta = \theta - \epsilon_k \times g / \sqrt{s+eps} $$

• RMSProp modifies AdaGrad by changing the gradient accumulation into an exponentially weighted moving average, it discards history from the extreme past. RMSProp has been shown to be an effective and practical optimization algorithm for deep neural networks.

$$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ s = \mathrm{decay\_rate}\times s + (1-\mathrm{decay\_rate}) g^{T}g \\ \theta = \theta - \epsilon_k \times g / \sqrt{s+eps} $$

• Adam derives from "adaptive moments", it can be seen as a variant on the combination of RMSProp and momentum, the update looks like RMSProp except that a smooth version of the gradient is used instead of the raw stochastic gradient, the full adam update also includes a bias correction mechanism, recommended values in the paper are $\epsilon = 1e-8$, $\beta_1 = 0.9$, $\beta_2 = 0.999$ $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ m = \beta_1 m + (1-\beta_1) g\\ s = \beta_2 v + (1-\beta_2) g^{T}g \\ \theta = \theta - \epsilon_k \times m / \sqrt{s+eps} $$

For more information about the algorithms, I recommend referring to deep learning book by Ian Goodfellow and a neural network course.

Here's a list of popular algorithms used in training neural networks:

• SGD updates parameters in the negative direction of the gradient, learning rate parameter $\epsilon_k$ should be decreased over time, computation time is proportional to mini-batch size $m$: $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ \theta = \theta - \epsilon_k \times g $$

• Momentum accumulates exponentially decaying moving average of past gradients and continues to move in their direction, thus step size depends on how large and how aligned the sequence of gradients are, common values of momentum parameter $\alpha$ are 0.5 and 0.9 $$ v = \alpha v - \epsilon \nabla_\theta \big(\frac{1}{m}\sum_i L(f(x^{(i)};\theta), y^{(i)}) \big) \\ \theta = \theta + v $$

• Nesterov Momentum inspired by Nesterov's accelerated gradient method, difference betweent Nesterov and standard momentum is where the gradient is evaluated, with Nesterov's momentum the gradient is evaluated after the current velocity is applied, thus Nesterov's momentum adds a correction factor to the gradient: $$ v = \alpha v - \epsilon \nabla_\theta \big(\frac{1}{m}\sum_i L(f(x^{(i)};\theta + \alpha \times v), y^{(i)}) \big) \\ \theta = \theta + v $$

• AdaGrad adapts the learning rates of all model parameters, learning rate is inversely proportional to the square root of the sum of historical squared values, weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased: the net effect is greater progress in the more gently sloped directions of parameter space: $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ s = s + g^{T}g \\ \theta = \theta - \epsilon_k \times g / \sqrt{s+eps} $$

• RMSProp modifies AdaGrad by changing the gradient accumulation into an exponentially weighted moving average, it discards history from the extreme past. RMSProp has been shown to be an effective and practical optimization algorithm for deep neural networks.

$$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ s = \mathrm{decay\_rate}\times s + (1-\mathrm{decay\_rate}) g^{T}g \\ \theta = \theta - \epsilon_k \times g / \sqrt{s+eps} $$

• Adam derives from "adaptive moments", it can be seen as a variant on the combination of RMSProp and momentum, the update looks like RMSProp except that a smooth version of the gradient is used instead of the raw stochastic gradient, the full adam update also includes a bias correction mechanism, recommended values in the paper are $\epsilon = 1e-8$, $\beta_1 = 0.9$, $\beta_2 = 0.999$ $$ g = \frac{1}{m}\nabla_\theta \sum_i L(f(x^{(i)};\theta), y^{(i)}) \\ m = \beta_1 m + (1-\beta_1) g\\ s = \beta_2 v + (1-\beta_2) g^{T}g \\ \theta = \theta - \epsilon_k \times m / \sqrt{s+eps} $$

For more information about the algorithms, I recommend referring to deep learning book by Ian Goodfellow and a neural network course. For a TensorFlow implementation have a look at the following ipython notebook.