Adam (adaptive) optimizer(s) learning rate tuning I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate:

The learning is arguably the most important parameter. In general, the
optimal learning rate is about half of the maximum learning rate (i.e.
the learning rate above which the training algorithm diverges, as we
saw in Chapter 4). One way to find a good learning rate is to train
the model for a few hundred iterations, starting with a very low
learning rate (e.g., 1e-5) and gradually increasing it up to a very
large value (e.g., 10). This is done by multiplying the learning rate
by a constant factor at each iteration (e.g., by exp(1e6/500) to go
from 1e-5 to 10 in 500 iterations). If you plot the loss as a function
of the learning rate (using log scale for a learning rate), you should
see it dropping at first. But after a while, the learning rate will be
too large, so the loss will shoot back up: the optimal learning rate
will be a bit lower than the point at which the loss starts to climb
(typically about 10 times lower than the turning point). You can then
reinitialize your model and train it normally using this good learning
rate. (...)

My question is: does it apply to any group of optimizers or to SGD in particular?
 A: Adam is an adaptive algorithm, so it self-tunes during the training. In many cases you would get away with the default hyperparameters and they would not need tuning. As you can learn from this thread sometimes tuning the learning rate may lead to improvements, but also the range of known best values is smaller as compared to other algorithms. However it should usually not be your first concern. Also notice that for $\beta_1$ and $\beta_2$ hyperparameters the general advice is not to change the defaults, you should do it only when you have a good reason for that.
A: I've done some experimenting. I've used this tutorial, so objects like models or generators refer to the concepts from it and need to be implemented. I've tested for 4 models in total (linear, dense, conv and lstm), each with 2 optimizers (adam and SGD). Below is a screenshot from tensorboard that  all the combinations behaved in the same way.

Below is the code I've used.
import math
import typing
from datetime import datetime
from itertools import product

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import ticker
from tensorflow import keras
from tensorflow.keras import callbacks, optimizers, losses

from tftst import models, window
from tftst.config import PLOTS_DIR, LOGS_DIR

START_LR = 1e-8
END_LR = 1e-1
STEP = 20
EPOCHS = int(math.log(END_LR / START_LR, 10)) * STEP + STEP


def adam() -> optimizers.Adam:
    return optimizers.Adam(START_LR)


def sgd() -> optimizers.SGD:
    return optimizers.SGD(START_LR, momentum=0.9)


def tune(m_producer: typing.Callable[[], keras.Model], o_producer: typing.Callable[[], optimizers.Optimizer], gen: window.WindowGenerator):
    model = m_producer()
    print(f'Tuning LR for model: {model.name}')

    optimizer = o_producer()
    optimizer_cls = type(optimizer).__name__

    current_time = datetime.now().strftime("%Y%m%d-%H%M%S")
    logs = LOGS_DIR / f'lr-tuning-{model.name}-{optimizer_cls}-{current_time}'

    model.compile(loss=losses.MeanSquaredError(), optimizer=optimizer)
    history = model.fit(gen.train, epochs=EPOCHS, callbacks=[
        callbacks.LearningRateScheduler(lambda epoch: START_LR * 10 ** (epoch / STEP)),
        callbacks.TensorBoard(str(logs), write_graph=False),
    ])

    plt.semilogx(history.history['lr'], history.history['loss'])
    plt.axis([START_LR, END_LR * 1e1, 0, 3])

    ax = plt.gca()
    x_major = matplotlib.ticker.LogLocator(base=10.0, numticks=5)
    ax.xaxis.set_major_locator(x_major)
    x_minor = matplotlib.ticker.LogLocator(base=10.0, subs=np.arange(1.0, 10.0) * 0.1, numticks=10)
    ax.xaxis.set_minor_locator(x_minor)
    ax.xaxis.set_minor_formatter(matplotlib.ticker.NullFormatter())

    img = PLOTS_DIR / f'{model.name}-{optimizer_cls}-lr-tuning.png'
    plt.savefig(img)

    plt.title(f'{model.name}-{optimizer_cls}')
    plt.show()


if __name__ == '__main__':
    print('Tuning started')
    print(f'Start LR: {START_LR}')
    print(f'End LR: {END_LR}')
    print(f'Epochs LR: {EPOCHS}')
    print(f'Step: {STEP}')

    ww = window.WIDE_WINDOW
    wcw = window.WIDE_CONV_WINDOW

    for opt, tup in product([sgd, adam],
                            zip(
                                [models.linear, models.dense, models.conv, models.lstm],
                                [ww, ww, wcw, ww]
                            )):
        mod = tup[0]
        win = tup[1]
        tune(mod, opt, win)

