47
$\begingroup$

Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations:

$$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t),$$ where $E({\bf w})$ is the error function, ${\bf w}$ - the vector of weights, $\eta$ - learning rate.

Weight decay $\lambda$ penalizes the weight changes:

$$\Delta\omega_i(t+1) =- \eta\frac{\partial E}{\partial w_i} - \lambda\eta\omega_i$$

The question is if it makes sense to combine both tricks during the back-propagation and what effect it would have?

$$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t) - \lambda\eta\omega_i$$

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Do you mean to say ωi(t+1) = ωi − η ∂E/∂wi + αΔωi(t), instead of Δωi(t+1) = ωi − η∂E/∂wi + αΔωi(t) ? $\endgroup$
    – hakunamatata
    Apr 9, 2016 at 22:42

1 Answer 1

Reset to default
51
$\begingroup$

Yes, it's very common to use both tricks. They solve different problems and can work well together.

One way to think about it is that weight decay changes the function that's being optimized, while momentum changes the path you take to the optimum.

Weight decay, by shrinking your coefficients toward zero, ensures that you find a local optimum with small-magnitude parameters. This is usually crucial for avoiding overfitting (although other kinds of constraints on the weights can work too). As a side benefit, it can also make the model easier to optimize, by making the objective function more convex.

Once you have an objective function, you have to decide how to move around on it. Steepest descent on the gradient is the simplest approach, but you're right that fluctuations can be a big problem. Adding momentum helps solve that problem. If you're working with batch updates (which is usually a bad idea with neural networks) Newton-type steps are another option. The new "hot" approaches are based on Nesterov's accelerated gradient and so-called "Hessian-Free" optimization.

But regardless of which of these update rules you use (momentum, Newton, etc.), you're still working with the same objective function, which is determined by your error function (e.g. squared error) and other constraints (e.g. weight decay). The main question when deciding which of these to use is how quickly you'll get to a good set of weights.

Improve this answer
$\endgroup$
4
  • $\begingroup$ 'it can also make the model easier to optimize, by making the objective function more convex' - could you please explain how smaller weights make this possible? $\endgroup$
    – Alex
    Aug 16, 2017 at 11:15
  • $\begingroup$ Here's a simple example that should illustrate the point: if your original objective function is $\mathrm{sin}(x)$, there are infinitely many local minima. If you add $a x^2$ to the objective function, and $a$ is bigger than about 0.2, all of the old local optima will disappear and you'll be left with just one minimum near 0. $\endgroup$
    – David J. Harris
    Aug 16, 2017 at 21:36
  • $\begingroup$ Nice answer, thank you. What about Adam Optimizer ? Does it perform better that the combination of weight decay and momentum ? $\endgroup$
    – A. Piro
    Nov 23, 2017 at 15:10
  • $\begingroup$ Adam is like momentum, but not like weight decay; it affects how you navigate the objective function, but not the objective function itself. $\endgroup$
    – David J. Harris
    Nov 24, 2017 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.