Sigmoid functions that are symmetric about the origin have been recommended in neural nets literature (for instance, Efficient Backprop by LeCun et al.). Hence, a tanh activation function with a range between -1 and 1 is preferred over a logistic sigmoid with a range between 0 and 1. To save the logistic sigmoid, why are its outputs not normalized (zero mean and unit variance)?
$\begingroup$
$\endgroup$
5
-
1$\begingroup$ What do you mean with normalized? Do you mean the activated distribution, or just the function itself? I don't understand your question. Sigmoid and tanh are closely related $tanh( x)=2⋅σ(2x)−1$. $\endgroup$– Harald ThomsonCommented Jul 23, 2017 at 10:42
-
$\begingroup$ By normalizing, I mean shifting and scaling the output values in a particular layer so that they have zero mean and unit variance. So, this would be the activated distribution and not the function. $\endgroup$– niranjantdesaiCommented Jul 23, 2017 at 12:52
-
$\begingroup$ The activated distribution obviously depends in your data. Shifting is easy by an additional bias term. Variance normalization can be done by batch normalization. $\endgroup$– Harald ThomsonCommented Jul 23, 2017 at 12:55
-
$\begingroup$ My question is why is it not typically done? $\endgroup$– niranjantdesaiCommented Jul 24, 2017 at 11:19
-
2$\begingroup$ Well, it usually is. Neural network tend to work better when activations have zero mean and unit variance. That's the reason why batchnorm is used. However, recently a activation function was discovered which automatically has this standardization property. See: arxiv.org/abs/1706.02515 $\endgroup$– Harald ThomsonCommented Jul 24, 2017 at 11:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
For all real values v, sigmoid(v) is always ∈ (0, 1), and probabilities of events are also always ∈ (0, 1). Thus, the sigmoid function used in the output layer is good for modeling probability values directly, and for this purpose (modeling probabilities) there's no need to normalize the output values.
If adding an extra layer before the sigmoid output layer only for shifting and scaling output values, then this does not improve performance but only adds computational cost. Since theoretically, linear operation, already reflected in weights of weight matrices, does not enhance model complexity.
