# Why is the default learning rate for Adadelta so low in Keras?

I have been training a model using the Adadelta optimizer for some time, and I noticed that it converges very, very slowly. Then I checked the Keras documentation, and to my surprise the default learning rate is 0.001.

This is 1000 times smaller than the learning rate of the "real" Adadelta optimizer. When I set it to 1, my model converged significantly faster. Why has Keras chosen to set the rate so low by default?

• I would guess that it is just a conservative value to prevent overshooting Oct 28 '20 at 13:15
• @JavierTG could be, but that's very model / problem specific. Oct 28 '20 at 14:25

If you look into TF source code you will find

def __init__(self, learning_rate=0.001, rho=0.95, epsilon=1e-8,
Args:
learning_rate: A Tensor or a floating point value. The learning rate.
To match the exact form in the original paper use 1.0.
rho: A Tensor or a floating point value. The decay rate.
epsilon: A Tensor or a floating point value.  A constant epsilon used
to better conditioning the grad update.
use_locking: If True use locks for update operations.
name: Optional name prefix for the operations created when applying
[...]


Developers are aware that the paper used learning_rate=1.0. They put that notice there due to this issue.

With the exception of SGD, all other major optimizers have learning_rate=0.001, so it probably got the same value by coincidence.

It seems like Keras has enforced the same default values for each of the different optimizers. For most of the optimizers listed on this page, i.e.

• RMSprop
the default learning rate is always set to 0.001.