# How to (systematically) tune learning rate having Gradient Descent as the Optimizer?

An outsider to ML/DL field; started Udacity Deep Learning course which is based on Tensorflow; doing the assignment 3 problem 4; trying to tune the learning rate with the following config:

• Batch size 128
• Number of steps: enough to fill up 2 epochs
• Sizes of hidden layers: 1024, 305, 75
• Weight initialization: truncated normal with std. deviation of sqrt(2/n) where n is the size of the previous layer
• Dropout keep probability: 0.75
• Regularization: not applied
• Learning Rate algorithm: exponential decay

played around with learning rate parameters; they don't seem to have effect in most cases; code here; results:

Accuracy    learning_rate   decay_steps     decay_rate      staircase
93.7        .1              3000            .96             True
94.0        .3              3000            .86             False
94.0        .3              3000            .96             False
94.0        .3              3000            .96             True
94.0        .5              3000            .96             True

• How should I systematically tune learning rate?
• How is learning rate related to the number of steps?
• tpot provides automatic ML tuning pipelines – denfromufa May 11 '16 at 14:52

## 4 Answers

Use a gradient descent optimizer. This is a very good overview.

Regarding the code, have a look at this tutorial. This and this are some examples.

Personally, I suggest to use either ADAM or RMSprop. There are still some hyperparameters to set, but there are some "standard" ones that work 99% of the time. For ADAM you can look at its paper and for RMSprop at this slides.

EDIT

Ok, you already use a gradient optimizer. Then you can perform some hyperparameters optimization to select the best learning rate. Recently, an automated approach has been proposed. Also, there is a lot of promising work by Frank Hutter regarding automated hyperparameters tuning.

More in general, have a look at the AutoML Challenge, where you can also find source code by the teams. In this challenge, the goal is to automate machine learning, including hyperparameters tuning.

Finally, this paper by LeCun and this very recent tutorial by DeepMin (check Chapter 8) give some insights that might be useful for your question.

Anyway, keep in mind that (especially for easy problems), it's normal that the learning rate doesn't affect much the learning when using a gradient descent optimizer. Usually, these optimizers are very reliable and work with different parameters.

• I am already using Gradient Descent Optimizer in the code. Thanks for the links! – Thoran Apr 20 '16 at 13:58
• @Thoran Ok, didn't read the code :D (and the question doesn't mention an optimizer). I have edited my answer to give you some more help :) – Simon Apr 22 '16 at 7:10
• Nice developments, makes the job easier for outsiders like <. Do you by any know how number of steps and learning rate are related? My gut tells me that if there are more steps, the learning process should be slower. – Thoran Apr 22 '16 at 7:39
• @Thoran Typically yes, the higher the number of steps, the slower the process (but also the more stable). I further edited my answer by adding some references for some "tricks" that can be useful for hand tuning the hyperparameters. – Simon Apr 22 '16 at 11:16
• very cool stuff, it will take some time for me to digest it :D – Thoran Apr 22 '16 at 11:41

You can automate the tuning of hyper-parameters in a lot of machine learning algorithms themselves, or just the hyperparameters for Gradient Descent optimizer i.e learning rate.

One library that has been popular for doing this is spearmint.

https://github.com/JasperSnoek/spearmint

A very recent automatic learning-rate tuner is given in Online Learning Rate Adaptation with Hypergradient Descent

This method is very straightforward to implement, the core result for SGD is given as:

$\alpha_{t} = \alpha_{t-1} + \beta \nabla f(\theta_{t-1})^T\nabla f(\theta_{t-2})$

where $\beta$ is a (hyper) hyperparameter. The method also applies to other gradient-based updates ($\textit{e.g.}$ momentum-based methods). No validation set is needed: it only requires storing the previous gradient, $\nabla f(\theta_{t-2})$. The idea is to use the partial derivative of the objective function w.r.t. the learning rate ($\alpha$), to derive an update rule for alpha.

Anecdotally, I implemented this on top of my existing problem, and observed much better results. I did not tune $\beta$ or $\alpha_0$, but picked from the suggested ranges from the paper.

To tune hyperparameters (whether it is learning rate, decay rate, regularization, or anything else), you need to establish a heldout dataset; this dataset is disjoint from your training dataset. After tuning several models with different configurations (where a configuration = a particular choice of each hyperparameter), you choose the configuration by selecting the one that maximizes heldout accuracy.

• How is heldout dataset different from test or validation dataset? – Thoran Apr 22 '16 at 6:16
• heldout is another name for validation. Basically you have three disjoint datasets: train, heldout, and test. Train is only used for learning model weights (e.g., using gradient descent). Heldout is used for tuning model parameters. Test is only used measuring generalization performance (i.e. how can I expect my model to perform on unseen data) – Alireza Apr 22 '16 at 7:14
• Thanks @Alireza for taking your time to explain. I am using validation dataset already. And without test, it's not possible to measure accuracy! – Thoran Apr 22 '16 at 7:30
• Well, your original question was about determining learning rate: if the first column in your original post was heldout (not training) accuracy, then you can say 0.5 is better than 0.1 (though you should continue to try values > 0.5). [note: if you re-do your experiments, just partition off some data to serve as test] – Alireza Apr 22 '16 at 7:55