I am training a neural network in TensorFlow and I would like to use firstly an exponential decay optimizer scheduler (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/ExponentialDecay) and then also a cosine decay (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/CosineDecay).

I need to set the decay rate for exponential decay and alpha for cosine decay in such a way, that after x epochs, the initial learning rate will just pass the final learning rate value.

I am having trouble formalizing this mathematically and calculating the necessary decay rates.

Any ideas?


1 Answer 1


I ended up figuring it out.

For the exponential decay, it was easier than I thought, as the formula for the decay is

initial_learning_rate * decay_rate ^ (step / decay_steps)

and since by the end of the training, the exponent should be one, it is enough to determine the decay rate as follows:

decay_rate = final_learning_rate / initial_learning_rate

For the cosine decay, it was slightly more obscure, since the computation is:

cosine_decay = 0.5 * (1 + cos(pi * step / decay_steps))
decayed = (1 - alpha) * cosine_decay + alpha
return initial_learning_rate * decayed

meaning that cosine decay is actually 0.5 * (1 + cos(pi)) at the last step and alpha is therefore:

(final_learning_rate - initial_learning_rate * cosine_decay) / (initial_learning_rate - initial_learning_rate * cosine_decay)

This seems to be working as I expected, so we can consider the issue solved.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.