It is suggested to normalize data as 0 mean and 1 variance. Also, TanH considered better than Sigmoid activation function as it has 0 mean. Why 0 mean is important?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ $\tanh(0)=0$ but that's not the same thing as having a mean of zero. Suppose $x$ is uniformly distributed on $[-2,-1]$; in this case, $\mathbb{E}(\tanh(x)) < 0$. $\endgroup$– Sycorax ♦Commented Jul 4, 2019 at 19:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Normalization is required only when features have different ranges. Because different features have different ranges of values, gradients may end up taking a long time to converge. They can oscillate back and forth before they can find a way to the global/local minimum. To overcome this, we normalize the data.
The optimisation of the neural net is less eradict, since the hidden activation functions don't saturate as fast and thus, don't produce near zero gradients (exploding gradient) early on in learning.
So, I think it is the centering and scaling that is important, as opposed to the literal value of zero.
This question may be a duplicate
