4
$\begingroup$

I read this paper on the "Cyclical Learning Rate" method, which cyclically decreases and then INCREASES the learning rate in gradient descent: http://arxiv.org/abs/1506.01186

Can anyone point me to, or does anyone know of, cases where someone tried randomly changing the learning rate instead of using some prescribed learning rate update schedule?

For example, randomly choosing a learning rate uniformly within some bounds every N iterations, or drawing the rate from a normal distribution?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This should be a comment. I haven't tried the author approach, however, what I often do with streaming algorithms is training them multiple times with the same parameters on shuffled versions of the original data. This improves performance and the why is vastly documented. I guess that multiple epochs with random learning rates is equivalent to multiple shuffling and training. Note that it is specially interesting in the case of large files as you do not need to go through costly shufflings any more. $\endgroup$
    – RUser4512
    Commented Sep 11, 2015 at 12:13

1 Answer 1

Reset to default
2
$\begingroup$

I just want to add, after three years the question was raised, that there is a paper submitted to ICLR 2019:

LEARNING WITH RANDOM LEARNING RATES https://openreview.net/pdf?id=S1fcnoR9K7

The main idea of the paper is:

We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gradient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.