2
$\begingroup$

I have been training a model using the Adadelta optimizer for some time, and I noticed that it converges very, very slowly. Then I checked the Keras documentation, and to my surprise the default learning rate is 0.001.

This is 1000 times smaller than the learning rate of the "real" Adadelta optimizer. When I set it to 1, my model converged significantly faster. Why has Keras chosen to set the rate so low by default?

$\endgroup$
2
  • $\begingroup$ I would guess that it is just a conservative value to prevent overshooting $\endgroup$
    – Javier TG
    Oct 28 '20 at 13:15
  • 1
    $\begingroup$ @JavierTG could be, but that's very model / problem specific. $\endgroup$ Oct 28 '20 at 14:25
2
$\begingroup$

If you look into TF source code you will find

def __init__(self, learning_rate=0.001, rho=0.95, epsilon=1e-8,
               use_locking=False, name="Adadelta"):
    """Construct a new Adadelta optimizer.
    Args:
      learning_rate: A `Tensor` or a floating point value. The learning rate.
        To match the exact form in the original paper use 1.0.
      rho: A `Tensor` or a floating point value. The decay rate.
      epsilon: A `Tensor` or a floating point value.  A constant epsilon used
               to better conditioning the grad update.
      use_locking: If `True` use locks for update operations.
      name: Optional name prefix for the operations created when applying
        gradients.  Defaults to "Adadelta".
[...]

Developers are aware that the paper used learning_rate=1.0. They put that notice there due to this issue.

With the exception of SGD, all other major optimizers have learning_rate=0.001, so it probably got the same value by coincidence.

$\endgroup$
1
$\begingroup$

It seems like Keras has enforced the same default values for each of the different optimizers. For most of the optimizers listed on this page, i.e.

  • RMSprop
  • Adam
  • Adadelta
  • Adagrad
  • Adamax
  • Nadam
  • Ftrl

the default learning rate is always set to 0.001.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.