1
$\begingroup$

In the paper An overview of gradient descent optimization algorithms, the author discusses the Momentum algorithm:

SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another, which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum as in Image 2.

Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations as can be seen in Image 3. It does this by adding a fraction γ of the update vector of the past time step to the current update vector. enter image description here

On the other hand, when I read about the section Improving gradient descent through feature scaling in a book, it mentions,

Standardization shifts the mean of each feature so that it is centered at zero and each feature has a standard deviation of 1, i.e., $x^{(i)}=\frac{x^{(i)}-\mu_x}{\sigma_x}$. Feature standardization helps with gradient descent learning in that the optimizer has to go through fewer steps to find a good or optimal solution (the global cost minimum), as shown in the following figure.

The context of this section in the book is the ADALINE algorithm, whose optimization is much simpler than that of a neural network. enter image description here

So it seems to me that feature standardization can solve the problem of the SGD without Momentum, i.e., "SGD has trouble navigating ravines"? If this is true, it means we don't need the Momentum algorithm. I know this must be wrong. Could you tell me why feature scaling can not solve the motivation of Momentum? I confused myself.

Improve this question
$\endgroup$
  • $\begingroup$ The quote you take talks about the problem of SGD w/o the momentum. What is the problem of SGD with the momentum? $\endgroup$ – gunes Sep 13 '18 at 11:18
  • 1
    $\begingroup$ @gunes I think my question was not clear. I have re-edited the question. $\endgroup$ – CyberPlayerOne Sep 13 '18 at 11:31
  • 2
    $\begingroup$ @Tyler, I think your book is wrong ( with the image) [but don't have access to the book]. as I understand feature scaling is just rescaling w1 and w2 meaning that your slanted ellipse will still be a slanted ellipse after rescaling just the variance in w1 and w2 directions will be 1, (but not in any other direction eg diagonal). to have a circle you need to 'sphere' your data (see PCA) but this may be computationally too intensive ( especially in NN setting [where you have to do it at each layer - see batchnorm]) $\endgroup$ – seanv507 Sep 13 '18 at 12:22
  • $\begingroup$ @seanv507 Thanks for your comment. Could you explain more on "feature scaling is just rescaling w1 and w2 meaning that your slanted ellipse will still be a slanted ellipse after rescaling"? The effect of standardization is difficult for me to imagine. Can I image the process of standardization as squeezing the slanted ellipse with the original w1 and w2 into a small square centered at origin (0,0)? If this is Ok, then the original slanted ellipse will become a smaller slanted ellipse. $\endgroup$ – CyberPlayerOne Sep 14 '18 at 9:47
  • 1
    $\begingroup$ @Tyler yes exactly it squeezes the ellipse into a square.[and nn's are most often used for signal/image processing data where the input variables are correlated leading to 'very' slanted weight surface too] $\endgroup$ – seanv507 Sep 14 '18 at 11:46
2
$\begingroup$

First of all, both of these methods try to address similar problems, but from very different angles. The error ellipses in your figure (or in the document) could be just caused by unequal feature scales. In this case, feature normalization quickly solves this issue, and these error ellipses become more circular. However, this doesn't mean that normalized dimensions' error curves don't look like ellipses, and cannot have steep directions. That is addressed by the momentum approach.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.