2
$\begingroup$

Glorot & Bengio (2010) show the sigmoid activation function is problematic because it saturates for large inputs and loses much of its non-linearity for small inputs, and suggest using zero-centered activation functions and sampling from a scaled uniform distribution to initialize weights. Sigmoid activations can be avoided in hidden layers, but I don't see any way around using them in the output layer for binary classification. What is an effective way of initializing the weights of a sigmoid output layer?

$\endgroup$

1 Answer 1

3
$\begingroup$

The paper (and recommended initialization) only refers to the hidden layers. They themselves used a softmax (which is a generalized sigmoid) on the final output layer. If you follow their variance derivation you see that they keep the variance w.r.t. the final activation as a fixed term which they don't manipulate:

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.