Defining the learning rate in Neural Network / MLP

Question

Long story short, I have been reading some stuff on neural networks recently, and realized that the MLP algorithm depends on a parameter $\alpha(t)$ which is the learning rate. Does $\alpha(t)$ have a general functional form? I'm wondering about this because some of the texts write $\alpha$ as $\alpha(t)$, while others write simply $\alpha$ and assign it a fixed number like $0.01$ or $0.1$ etc...

By the way, I found this source online: http://users.ics.aalto.fi/jhollmen/dippa/node22.html

I don't know if the functional forms they provide for $\alpha(t)$ are applicable to feedforward neural networks.

Thanks. Ikjyot Singh Kohli

Answer 1

This is because the learning rate sometimes changes after a certain number of iterations.

http://cs231n.github.io/neural-networks-3/#anneal gives more details:

In training deep networks, it is usually helpful to anneal the learning rate over time. Good intuition to have in mind is that with a high learning rate, the system contains too much kinetic energy and the parameter vector bounces around chaotically, unable to settle down into deeper, but narrower parts of the loss function. Knowing when to decay the learning rate can be tricky: Decay it slowly and you'll be wasting computation bouncing around chaotically with little improvement for a long time. But decay it too aggressively and the system will cool too quickly, unable to reach the best position it can. There are three common types of implementing the learning rate decay:

• Step decay: Reduce the learning rate by some factor every few epochs. Typical values might be reducing the learning rate by a half every 5 epochs, or by 0.1 every 20 epochs. These numbers depend heavily on the type of problem and the model. One heuristic you may see in practice is to watch the validation error while training with a fixed learning rate, and reduce the learning rate by a constant (e.g. 0.5) whenever the validation error stops improving.
• Exponential decay. has the mathematical form $\alpha = \alpha_0 e^{-k t}$, where $\alpha_0, k$ are hyperparameters and $t$ is the iteration number (but you can also use units of epochs).
• 1/t decay has the mathematical form $\alpha = \alpha_0 / (1 + k t )$ where $a_0, k$ are hyperparameters and $t$ is the iteration number.

In practice, we find that the step decay dropout is slightly preferable because the hyperparameters it involves (the fraction of decay and the step timings in units of epochs) are more interpretable than the hyperparameter $k$. Lastly, if you can afford the computational budget, err on the side of slower decay and train for a longer time.

Note that some parameter update policies such as Adagrad, RMSprop, or Adam don't require to set a learning rate.

Answer 2

Frank is correct. There are many ways to define a learning rate. It can be a function, or it can just be a fixed number. Most of the materials that you see on the Internet use a fixed number because it's simpler to train and interpret. There is no right or wrong on what methods to use, the only way is to train the model and evaluate the time and convergence.