Questions tagged [nesterov]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
56 views

How should I set $\vec{\mu}$ in NAdam optimization?

In Dozat 2016 they introduce a sequence of hyperparameters $\mu_0, \cdots, \mu_T$ where $T$ is the total number of iterations. Naturally $T$ is dependent on the convergence of the parameters, so it ...
Galen's user avatar
  • 8,095
2 votes
0 answers
1k views

Nesterov's Momentum (or Accelerated Gradient)

This might be a silly question, but here it is anyway. I'm trying to implement Nesterov's Momentum to extend the gradient descent algorithm that I'm currently using for my neural network, where I'm ...
Francesco Cariaggi's user avatar
1 vote
1 answer
1k views

Trying to write Nesterov Optimization - Gradient Descent

Problem: Im unsure if I understood Nesterov Optimization Im writing about Nesterov Optimization, but the notation im using seems different from the references below. I have done it using some books ...
KenobiBastila's user avatar
15 votes
1 answer
18k views

How to choose between SGD with Nesterov momentum and Adam?

I'm currently implementing a neural network architecture on Keras. I would like to optimize the training time, and I'm considering using alternative optimizers such as SGD with Nesterov Momentum and ...
Clément F's user avatar
  • 1,797