As stated in this thread Why do we need to normalize the images before we put them into CNN?, normalization of the inputs to a deep network to have zero mean and unit-deviation will make SGD more effective as each channel is on the same scale and thus the learning rate has the same effect in all.

This issue is though not encountered when using an optimizer that adjusts a learning rate specific to each weight in the model. So if a channel input is in a very different scale, the learning rate will just change proportional to that gradient landscape.

Is the performance of a model with normalized inputs using a vanilla SGD (with a scalar learning rate) expected to be the same as of a model with unnormalized inputs but using an adaptive learning rate optimizer, such as adam?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.