While creating a neural network in R, is it important to normalize the input data based on activation function? For Example - if the activation function is tanh input data should range from -1 to 1 and for sigmoid activation 0 to 1.


Your proposed normalizing scheme appears to try to normalize the inputs on the basis of the range of outputs for an activation function. This doesn't make much sense, since typical activation functions take any real number as an input.

I'm not aware of this method being used. Doing so seems to inhibit the outputs of common activations.

The sigmoid function $g$ gives outputs in $[0.5,1)$ for $x > 0$. If you normalize your inputs to be positive, you're giving up half the output rage of $g$.

The $\tanh$ function gives outputs in $[-0.76, 0.76]$ (approximately) for $x\in[-1,1]$. Rescaling the inputs to $\tanh$ in this way gives up almost 24% of the output range.

On the other hand, layer normalization is a popular option to re-scale the inputs to the next layer of the neural network. It has nice properties both in terms of regularization and speeding up training.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy