I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate:

The learning is arguably the most important parameter. In general, the optimal learning rate is about half of the maximum learning rate (i.e. the learning rate above which the training algorithm diverges, as we saw in Chapter 4). One way to find a good learning rate is to train the model for a few hundred iterations, starting with a very low learning rate (e.g., 1e-5) and gradually increasing it up to a very large value (e.g., 10). This is done by multiplying the learning rate by a constant factor at each iteration (e.g., by exp(1e6/500) to go from 1e-5 to 10 in 500 iterations). If you plot the loss as a function of the learning rate (using log scale for a learning rate), you should see it dropping at first. But after a while, the learning rate will be too large, so the loss will shoot back up: the optimal learning rate will be a bit lower than the point at which the loss starts to climb (typically about 10 times lower than the turning point). You can then reinitialize your model and train it normally using this good learning rate. (...)

My question is: does it apply to any group of optimizers or to SGD in particular?


2 Answers 2


Adam is an adaptive algorithm, so it self-tunes during the training. In many cases you would get away with the default hyperparameters and they would not need tuning. As you can learn from this thread sometimes tuning the learning rate may lead to improvements, but also the range of known best values is smaller as compared to other algorithms. However it should usually not be your first concern. Also notice that for $\beta_1$ and $\beta_2$ hyperparameters the general advice is not to change the defaults, you should do it only when you have a good reason for that.

  • $\begingroup$ This is poor advice -- generally LR is regarded as one of the most important parameters to tune, especially for larger models. It makes a massive difference even for optimizers like adam. The beta parameters should be tuned as well. Ignoring those three parameters will almost certainly leave performance on the table. Obviously getting the model into a reasonable configuration is the first priority, but once everything is settled tuning LR/Betas is very important. Also, one can do a LR sweep over several training runs on a small model, then adapt the LRs accordingly for a scaled up model. $\endgroup$
    – Arman
    Commented Jun 8 at 8:22

I've done some experimenting. I've used this tutorial, so objects like models or generators refer to the concepts from it and need to be implemented. I've tested for 4 models in total (linear, dense, conv and lstm), each with 2 optimizers (adam and SGD). Below is a screenshot from tensorboard that all the combinations behaved in the same way.

lr tuning chart

Below is the code I've used.

import math
import typing
from datetime import datetime
from itertools import product

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import ticker
from tensorflow import keras
from tensorflow.keras import callbacks, optimizers, losses

from tftst import models, window
from tftst.config import PLOTS_DIR, LOGS_DIR

START_LR = 1e-8
END_LR = 1e-1
STEP = 20
EPOCHS = int(math.log(END_LR / START_LR, 10)) * STEP + STEP

def adam() -> optimizers.Adam:
    return optimizers.Adam(START_LR)

def sgd() -> optimizers.SGD:
    return optimizers.SGD(START_LR, momentum=0.9)

def tune(m_producer: typing.Callable[[], keras.Model], o_producer: typing.Callable[[], optimizers.Optimizer], gen: window.WindowGenerator):
    model = m_producer()
    print(f'Tuning LR for model: {model.name}')

    optimizer = o_producer()
    optimizer_cls = type(optimizer).__name__

    current_time = datetime.now().strftime("%Y%m%d-%H%M%S")
    logs = LOGS_DIR / f'lr-tuning-{model.name}-{optimizer_cls}-{current_time}'

    model.compile(loss=losses.MeanSquaredError(), optimizer=optimizer)
    history = model.fit(gen.train, epochs=EPOCHS, callbacks=[
        callbacks.LearningRateScheduler(lambda epoch: START_LR * 10 ** (epoch / STEP)),
        callbacks.TensorBoard(str(logs), write_graph=False),

    plt.semilogx(history.history['lr'], history.history['loss'])
    plt.axis([START_LR, END_LR * 1e1, 0, 3])

    ax = plt.gca()
    x_major = matplotlib.ticker.LogLocator(base=10.0, numticks=5)
    x_minor = matplotlib.ticker.LogLocator(base=10.0, subs=np.arange(1.0, 10.0) * 0.1, numticks=10)

    img = PLOTS_DIR / f'{model.name}-{optimizer_cls}-lr-tuning.png'


if __name__ == '__main__':
    print('Tuning started')
    print(f'Start LR: {START_LR}')
    print(f'End LR: {END_LR}')
    print(f'Epochs LR: {EPOCHS}')
    print(f'Step: {STEP}')

    ww = window.WIDE_WINDOW
    wcw = window.WIDE_CONV_WINDOW

    for opt, tup in product([sgd, adam],
                                [models.linear, models.dense, models.conv, models.lstm],
                                [ww, ww, wcw, ww]
        mod = tup[0]
        win = tup[1]
        tune(mod, opt, win)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.