I have read in the literature and various other questions (as below) that normalising the inputs to have a mean of 0 and a standard deviation of 1 helps the gradient descent optimiser by making a steaper error surface.

However, do the inputs need to be normalised by taking into account the activation function in the network hidden layers? For example, when using sigmoid should the inputs be [0,1]? For tanh, should the inputs be [-1,1]? For relu, is it ok to have negative inputs?

Input Normalisation for ReLU neurons

Rescaling input features for neural networks regression

What characteristics should the input data have for a neural network?

Why normalise the standard deviation of neural network input?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.