How to calculate the decay rate given an initial learning rate and final learning rate for schedulers when training neural networks?

Question

I am training a neural network in TensorFlow and I would like to use firstly an exponential decay optimizer scheduler (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/ExponentialDecay) and then also a cosine decay (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/CosineDecay).

I need to set the decay rate for exponential decay and alpha for cosine decay in such a way, that after x epochs, the initial learning rate will just pass the final learning rate value.

I am having trouble formalizing this mathematically and calculating the necessary decay rates.

Any ideas?

Answer 1

I ended up figuring it out.

For the exponential decay, it was easier than I thought, as the formula for the decay is

initial_learning_rate * decay_rate ^ (step / decay_steps)

and since by the end of the training, the exponent should be one, it is enough to determine the decay rate as follows:

decay_rate = final_learning_rate / initial_learning_rate

For the cosine decay, it was slightly more obscure, since the computation is:

cosine_decay = 0.5 * (1 + cos(pi * step / decay_steps))
decayed = (1 - alpha) * cosine_decay + alpha
return initial_learning_rate * decayed

meaning that cosine decay is actually 0.5 * (1 + cos(pi)) at the last step and alpha is therefore:

(final_learning_rate - initial_learning_rate * cosine_decay) / (initial_learning_rate - initial_learning_rate * cosine_decay)

This seems to be working as I expected, so we can consider the issue solved.