In the context of neural networks, what is the difference between the learning rate and weight decay?


The learning rate is a parameter that determines how much an updating step influences the current value of the weights. While weight decay is an additional term in the weight update rule that causes the weights to exponentially decay to zero, if no other update is scheduled.

So let's say that we have a cost or error function $E(\mathbf{w})$ that we want to minimize. Gradient descent tells us to modify the weights $\mathbf{w}$ in the direction of steepest descent in $E$: \begin{equation} w_i \leftarrow w_i-\eta\frac{\partial E}{\partial w_i}, \end{equation} where $\eta$ is the learning rate, and if it's large you will have a correspondingly large modification of the weights $w_i$ (in general it shouldn't be too large, otherwise you'll overshoot the local minimum in your cost function).

In order to effectively limit the number of free parameters in your model so as to avoid over-fitting, it is possible to regularize the cost function. An easy way to do that is by introducing a zero mean Gaussian prior over the weights, which is equivalent to changing the cost function to $\widetilde{E}(\mathbf{w})=E(\mathbf{w})+\frac{\lambda}{2}\mathbf{w}^2$. In practice this penalizes large weights and effectively limits the freedom in your model. The regularization parameter $\lambda$ determines how you trade off the original cost $E$ with the large weights penalization.

Applying gradient descent to this new cost function we obtain: \begin{equation} w_i \leftarrow w_i-\eta\frac{\partial E}{\partial w_i}-\eta\lambda w_i. \end{equation} The new term $-\eta\lambda w_i$ coming from the regularization causes the weight to decay in proportion to its size.

  • 2
    $\begingroup$ Thanks for the useful explanation. A question: in the "nnet" R package there is a parameter used in the training of the neural network called "decay". Do you know if decay corresponds to your lambda or to your eta*lambda? $\endgroup$ – Andrea Ianni ௫ Oct 21 '15 at 12:50
  • 3
    $\begingroup$ I would also add that weight decay is the same thing as L2 regularization for those who are familiar the the latter. $\endgroup$ – Sergei Mar 26 '18 at 21:14
  • 3
    $\begingroup$ @Sergei please no, stop spreading this misinformation! This is only true in the very special case of vanilla SGD. See the Fixing weight decay for Adam paper. $\endgroup$ – LucasB Jun 14 '18 at 7:13
  • 1
    $\begingroup$ To clarify: at time of writing, the PyTorch docs for Adam uses the term "weight decay" (parenthetically called "L2 penalty") to refer to what I think those authors call L2 regulation. If I understand correctly, this answer refers to SGD without momentum, where the two are equivalent. $\endgroup$ – Dylan F Jun 15 '18 at 3:51

In addition to @mrig's answer (+1), for many practical application of neural networks it is better to use a more advanced optimisation algorithm, such as Levenberg-Marquardt (small-medium sized networks) or scaled conjugate gradient descent (medium-large networks), as these will be much faster, and there is no need to set the learning rate (both algorithms essentially adapt the learning rate using curvature as well as gradient). Any decent neural network package or library will have implementations of one of these methods, any package that doesn't is probably obsolete. I use the NETLAB libary for MATLAB, which is a great piece of kit.

  • $\begingroup$ What about the Rprop based optimisation algos? How do they stack up? $\endgroup$ – power Nov 14 '14 at 4:51
  • 4
    $\begingroup$ I think most people are using variants of rprop+momentum such as RMSProp, adagrad, SGD+nesterov momentum. See cs231 class. $\endgroup$ – Pepe Mandioca Dec 20 '16 at 18:33
  • 1
    $\begingroup$ Well, of course it depends on your application. But for large datasets/networks which are kinda trendy right now, I think people are finding those algorithms I mentioned better suited. $\endgroup$ – Pepe Mandioca Dec 24 '16 at 18:41
  • 8
    $\begingroup$ @DikranMarsupial it might be because nowadays (almost five years after your answer) people tend to use Adam instead? $\endgroup$ – ComputerScientist May 22 '17 at 21:45
  • 2
    $\begingroup$ Indeed this answer is very outdated. These algorithms are not practical with the scale of models and datasets that are typical nowadays, and the most widely used packages, which are certainly not obsolete, lack these algorithms. $\endgroup$ – LucasB Jun 14 '18 at 7:10

In simple terms:

learning_rate: It controls how quickly or slowly a neural network model learns a problem.

See: https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/

weight_decay: Is a regularisation technique used to avoid over-fitting.

See: https://metacademy.org/graphs/concepts/weight_decay_neural_networks


So the answer given by @mrig is actually intuitively alright. But theoretically speaking what he has explained is L2 regularization. This was known as weight decay back in the day but now I think the literature is pretty clear about the fact. These two concepts have a subtle difference and learning this difference can give a better understanding on weight decay parameter. It's easier to understand once you identify the two as which is which.

Here I'll discuss about the two regularization techniques known as L2 regularization and decoupled wight decay.

In L2 regularization you directly make changes to the cost function. This can be shown as follows using the same terminology as in @mrig's answer. \begin{equation} \widetilde{E}(\mathbf{w})=E(\mathbf{w})+\frac{\lambda}{2}\mathbf{w}^2 \end{equation}

So once you take the gradient (as in SGD optimizer), this simplifies down to the following equation: \begin{equation} w_i \leftarrow w_i-\eta\frac{\partial E}{\partial w_i}-\eta\lambda w_i \end{equation}

\begin{equation} w_i \leftarrow (1-\eta\lambda) w_i-\eta\frac{\partial E}{\partial w_i} \end{equation}

However, in decoupled weight decay, you do not do any adjustments to the cost function directly.

For the same SGD optimizer weight decay can be written as:

\begin{equation} w_i \leftarrow (1-\lambda^\prime) w_i-\eta\frac{\partial E}{\partial w_i} \end{equation}

So there you have it. The difference of the two techniques in SGD is subtle. When $\lambda = \frac{\lambda^\prime}{\eta}$ the two equations become the same. On the contrary, it makes a huge difference in adaptive optimizers such as Adam. This is extensively explained in the literature I have attached.

About the learning rate, I think the other answers have given a nice explanation about that and further explanation is unnecessary at this point.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.