Adam (adaptive) optimizer(s) learning rate tuning

Question

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate:

The learning is arguably the most important parameter. In general, the optimal learning rate is about half of the maximum learning rate (i.e. the learning rate above which the training algorithm diverges, as we saw in Chapter 4). One way to find a good learning rate is to train the model for a few hundred iterations, starting with a very low learning rate (e.g., 1e-5) and gradually increasing it up to a very large value (e.g., 10). This is done by multiplying the learning rate by a constant factor at each iteration (e.g., by exp(1e6/500) to go from 1e-5 to 10 in 500 iterations). If you plot the loss as a function of the learning rate (using log scale for a learning rate), you should see it dropping at first. But after a while, the learning rate will be too large, so the loss will shoot back up: the optimal learning rate will be a bit lower than the point at which the loss starts to climb (typically about 10 times lower than the turning point). You can then reinitialize your model and train it normally using this good learning rate. (...)

My question is: does it apply to any group of optimizers or to SGD in particular?

Answer 1

Adam is an adaptive algorithm, so it self-tunes during the training. In many cases you would get away with the default hyperparameters and they would not need tuning. As you can learn from this thread sometimes tuning the learning rate may lead to improvements, but also the range of known best values is smaller as compared to other algorithms. However it should usually not be your first concern. Also notice that for $\beta_1$ and $\beta_2$ hyperparameters the general advice is not to change the defaults, you should do it only when you have a good reason for that.

Answer 2

I've done some experimenting. I've used this tutorial, so objects like models or generators refer to the concepts from it and need to be implemented. I've tested for 4 models in total (linear, dense, conv and lstm), each with 2 optimizers (adam and SGD). Below is a screenshot from tensorboard that all the combinations behaved in the same way.

Below is the code I've used.

import math
import typing
from datetime import datetime
from itertools import product

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import ticker
from tensorflow import keras
from tensorflow.keras import callbacks, optimizers, losses

from tftst import models, window
from tftst.config import PLOTS_DIR, LOGS_DIR

START_LR = 1e-8
END_LR = 1e-1
STEP = 20
EPOCHS = int(math.log(END_LR / START_LR, 10)) * STEP + STEP


def adam() -> optimizers.Adam:
    return optimizers.Adam(START_LR)


def sgd() -> optimizers.SGD:
    return optimizers.SGD(START_LR, momentum=0.9)


def tune(m_producer: typing.Callable[[], keras.Model], o_producer: typing.Callable[[], optimizers.Optimizer], gen: window.WindowGenerator):
    model = m_producer()
    print(f'Tuning LR for model: {model.name}')

    optimizer = o_producer()
    optimizer_cls = type(optimizer).__name__

    current_time = datetime.now().strftime("%Y%m%d-%H%M%S")
    logs = LOGS_DIR / f'lr-tuning-{model.name}-{optimizer_cls}-{current_time}'

    model.compile(loss=losses.MeanSquaredError(), optimizer=optimizer)
    history = model.fit(gen.train, epochs=EPOCHS, callbacks=[
        callbacks.LearningRateScheduler(lambda epoch: START_LR * 10 ** (epoch / STEP)),
        callbacks.TensorBoard(str(logs), write_graph=False),
    ])

    plt.semilogx(history.history['lr'], history.history['loss'])
    plt.axis([START_LR, END_LR * 1e1, 0, 3])

    ax = plt.gca()
    x_major = matplotlib.ticker.LogLocator(base=10.0, numticks=5)
    ax.xaxis.set_major_locator(x_major)
    x_minor = matplotlib.ticker.LogLocator(base=10.0, subs=np.arange(1.0, 10.0) * 0.1, numticks=10)
    ax.xaxis.set_minor_locator(x_minor)
    ax.xaxis.set_minor_formatter(matplotlib.ticker.NullFormatter())

    img = PLOTS_DIR / f'{model.name}-{optimizer_cls}-lr-tuning.png'
    plt.savefig(img)

    plt.title(f'{model.name}-{optimizer_cls}')
    plt.show()


if __name__ == '__main__':
    print('Tuning started')
    print(f'Start LR: {START_LR}')
    print(f'End LR: {END_LR}')
    print(f'Epochs LR: {EPOCHS}')
    print(f'Step: {STEP}')

    ww = window.WIDE_WINDOW
    wcw = window.WIDE_CONV_WINDOW

    for opt, tup in product([sgd, adam],
                            zip(
                                [models.linear, models.dense, models.conv, models.lstm],
                                [ww, ww, wcw, ww]
                            )):
        mod = tup[0]
        win = tup[1]
        tune(mod, opt, win)