0
$\begingroup$

It is suggested to normalize data as 0 mean and 1 variance. Also, TanH considered better than Sigmoid activation function as it has 0 mean. Why 0 mean is important?

$\endgroup$
  • $\begingroup$ $\tanh(0)=0$ but that's not the same thing as having a mean of zero. Suppose $x$ is uniformly distributed on $[-2,-1]$; in this case, $\mathbb{E}(\tanh(x)) < 0$. $\endgroup$ – Sycorax says Reinstate Monica Jul 4 at 19:05
0
$\begingroup$

Normalization is required only when features have different ranges. Because different features have different ranges of values, gradients may end up taking a long time to converge. They can oscillate back and forth before they can find a way to the global/local minimum. To overcome this, we normalize the data.

The optimisation of the neural net is less eradict, since the hidden activation functions don't saturate as fast and thus, don't produce near zero gradients (exploding gradient) early on in learning.

So, I think it is the centering and scaling that is important, as opposed to the literal value of zero.

This question may be a duplicate

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.