Why my weight Initialization saturates the activation function?

Question

I am trying to initialize the Hidden Layers of my Convolutional Neural Network with 7 hidden layers, all with ELU Activation with Sigmoid Activation at the end for binary classification.

For Initialization of hidden layers, I've followed He Initialization by equating the standard deviation to sqrt(2.0/n) where n is the number of neurons in the previous layer.

What I find quite disturbing is that this initialization tends to saturate the output for some of the observations in my mini-batch, whereas following the "naive approach" and setting all standard deviations to 0.01 from a Gaussian Distribution avoids saturation for all observations (all outputs coming from the Sigmoid are approximately 0.5, which is ideal for SGD).

My question is what is the cause of this saturation and what is the correct way of initialization? Is the "naive approach" for initialization flawed?

I've also read through related posts but am still puzzled.

Answer 1

What I observed empirically is that sampling from a truncated normal distribution rather than a standard normal distribution better prevents saturation at the output sigmoid. This might be due to a more constrained variance.