0
$\begingroup$

This question already has an answer here:

It is suggested to normalize data as 0 mean and 1 variance. Also, TanH considered better than Sigmoid activation function as it has 0 mean. Why 0 mean is important?

$\endgroup$

marked as duplicate by Reinstate Monica, kjetil b halvorsen, Peter Flom - Reinstate Monica Jul 5 at 14:03

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ $\tanh(0)=0$ but that's not the same thing as having a mean of zero. Suppose $x$ is uniformly distributed on $[-2,-1]$; in this case, $\mathbb{E}(\tanh(x)) < 0$. $\endgroup$ – Reinstate Monica Jul 4 at 19:05
0
$\begingroup$

Normalization is required only when features have different ranges. Because different features have different ranges of values, gradients may end up taking a long time to converge. They can oscillate back and forth before they can find a way to the global/local minimum. To overcome this, we normalize the data.

The optimisation of the neural net is less eradict, since the hidden activation functions don't saturate as fast and thus, don't produce near zero gradients (exploding gradient) early on in learning.

So, I think it is the centering and scaling that is important, as opposed to the literal value of zero.

This question may be a duplicate

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.