Defining the learning rate in Neural Network / MLP Long story short, I have been reading some stuff on neural networks recently, and realized that the MLP algorithm depends on a parameter $\alpha(t)$ which is the learning rate. Does $\alpha(t)$ have a general functional form? I'm wondering about this because some of the texts write $\alpha$ as $\alpha(t)$, while others write simply $\alpha$ and assign it a fixed number like $0.01$ or $0.1$ etc...
By the way, I found this source online: http://users.ics.aalto.fi/jhollmen/dippa/node22.html
I don't know if the functional forms they provide for $\alpha(t)$ are applicable to feedforward neural networks.
Thanks.
Ikjyot Singh Kohli
 A: This is because the learning rate sometimes changes after a certain number of iterations. 
http://cs231n.github.io/neural-networks-3/#anneal gives more details:

In training deep networks, it is usually helpful to anneal the
  learning rate over time. Good intuition to have in mind is that with a
  high learning rate, the system contains too much kinetic energy and
  the parameter vector bounces around chaotically, unable to settle down
  into deeper, but narrower parts of the loss function. Knowing when to
  decay the learning rate can be tricky: Decay it slowly and you'll be
  wasting computation bouncing around chaotically with little
  improvement for a long time. But decay it too aggressively and the
  system will cool too quickly, unable to reach the best position it
  can. There are three common types of implementing the learning rate
  decay:
  
  
*
  
*Step decay: Reduce the learning rate by some factor every few epochs. Typical values might be reducing the learning rate by a half
  every 5 epochs, or by 0.1 every 20 epochs. These numbers depend
  heavily on the type of problem and the model. One heuristic you may
  see in practice is to watch the validation error while training with a
  fixed learning rate, and reduce the learning rate by a constant (e.g.
  0.5) whenever the validation error stops improving.
  
*Exponential decay. has the mathematical form $\alpha = \alpha_0 e^{-k t}$, where $\alpha_0, k$ are hyperparameters and $t$ is the
  iteration number (but you can also use units of epochs).
  
*1/t decay has the mathematical form $\alpha = \alpha_0 / (1 + k t )$ where $a_0, k$ are hyperparameters and $t$ is the iteration
  number.
  
  
  In practice, we find that the step decay dropout is slightly
  preferable because the hyperparameters it involves (the fraction of
  decay and the step timings in units of epochs) are more interpretable
  than the hyperparameter $k$. Lastly, if you can afford the
  computational budget, err on the side of slower decay and train for a
  longer time.

Note that some parameter update policies such as Adagrad, RMSprop, or Adam don't require to set a learning rate.
A: Frank is correct. There are many ways to define a learning rate. It can be a function, or it can just be a fixed number. Most of the materials that you see on the Internet use a fixed number because it's simpler to train and interpret. There is no right or wrong on what methods to use, the only way is to train the model and evaluate the time and convergence.
