Sigmoid functions that are symmetric about the origin have been recommended in neural nets literature (for instance, Efficient Backprop by LeCun et al.). Hence, a tanh activation function with a range between -1 and 1 is preferred over a logistic sigmoid with a range between 0 and 1. To save the logistic sigmoid, why are its outputs not normalized (zero mean and unit variance)?

  • 1
    $\begingroup$ What do you mean with normalized? Do you mean the activated distribution, or just the function itself? I don't understand your question. Sigmoid and tanh are closely related $tanh( x)=2⋅σ(2x)−1$. $\endgroup$ – Harald Thomson Jul 23 '17 at 10:42
  • $\begingroup$ By normalizing, I mean shifting and scaling the output values in a particular layer so that they have zero mean and unit variance. So, this would be the activated distribution and not the function. $\endgroup$ – niranjantdesai Jul 23 '17 at 12:52
  • $\begingroup$ The activated distribution obviously depends in your data. Shifting is easy by an additional bias term. Variance normalization can be done by batch normalization. $\endgroup$ – Harald Thomson Jul 23 '17 at 12:55
  • $\begingroup$ My question is why is it not typically done? $\endgroup$ – niranjantdesai Jul 24 '17 at 11:19
  • 2
    $\begingroup$ Well, it usually is. Neural network tend to work better when activations have zero mean and unit variance. That's the reason why batchnorm is used. However, recently a activation function was discovered which automatically has this standardization property. See: arxiv.org/abs/1706.02515 $\endgroup$ – Harald Thomson Jul 24 '17 at 11:46

For all real values v, sigmoid(v) is always ∈ (0, 1), and probabilities of events are also always ∈ (0, 1). Thus, the sigmoid function used in the output layer is good for modeling probability values directly, and for this purpose (modeling probabilities) there's no need to normalize the output values.

If adding an extra layer before the sigmoid output layer only for shifting and scaling output values, then this does not improve performance but only adds computational cost. Since theoretically, linear operation, already reflected in weights of weight matrices, does not enhance model complexity.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.