In the paper An overview of gradient descent optimization algorithms, the author discusses the Momentum algorithm:
SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another, which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum as in
Image 2
.Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations as can be seen in
Image 3
. It does this by adding a fraction γ of the update vector of the past time step to the current update vector.
On the other hand, when I read about the section Improving gradient descent through feature scaling in a book, it mentions,
Standardization shifts the mean of each feature so that it is centered at zero and each feature has a standard deviation of 1, i.e., $x^{(i)}=\frac{x^{(i)}-\mu_x}{\sigma_x}$. Feature standardization helps with gradient descent learning in that the optimizer has to go through fewer steps to find a good or optimal solution (the global cost minimum), as shown in the following figure.
The context of this section in the book is the ADALINE algorithm, whose optimization is much simpler than that of a neural network.
So it seems to me that feature standardization can solve the problem of the SGD without Momentum, i.e., "SGD has trouble navigating ravines"? If this is true, it means we don't need the Momentum algorithm. I know this must be wrong. Could you tell me why feature scaling can not solve the motivation of Momentum? I confused myself.
"feature scaling is just rescaling w1 and w2 meaning that your slanted ellipse will still be a slanted ellipse after rescaling"?
The effect of standardization is difficult for me to imagine. Can I image the process of standardization as squeezing the slanted ellipse with the original w1 and w2 into a small square centered at origin (0,0)? If this is Ok, then the original slanted ellipse will become a smaller slanted ellipse. $\endgroup$ – CyberPlayerOne Sep 14 '18 at 9:47