I am a new to deep learning and neural networks, and I need to know if there is a good weights initialization method to use if the activation function is Softmax like Tanh, ReLU and Sigmoid. Related answer.
- ReLU and variants like PReLU, RReLU and ELU: use He initialization (uniform or normal)
- SELU: use LeCun initialization (normal) (see this paper)
- Default (including Sigmoid, Tanh, Softmax, or no activation): use Xavier initialization (uniform or normal), also called Glorot initialization. This is the default in Keras and most other deep learning libraries.
When initializing the weights with a normal distribution, all these methods use mean 0 and variance σ²=scale/fan_avg or σ²=scale/fan_in. The fan_in is the layer's number of inputs, the fan_out is the layer's number of outputs (=number of neurons), fan_avg is the average between the two =½(fan_in+fan_out). Specifically:
- Xavier: σ²=1/fan_avg
- He: σ²=2/fan_in
- LeCun: σ²=1/fan_in
When initializing the weights with a uniform distribution, all these methods just use the range [-limit, limit] where limit = sqrt(3 * σ²).
If you have consecutive ReLU layers with very different sizes, you may prefer using fan_avg rather than fan_in. In Keras, you can use something like this:
init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg', distribution='normal') layer = Dense(10, activation="relu", kernel_initializer=init)
He or Xavier Initialization is usually what is recommended. See for example this article:
I recommend simply remembering:
Use Xavier Initialization in the fully-connected layers of your network. (Or layers that use softmax/tanh activation functions)
Use Variance Scaling Initialization in the intermediate layer of your network that use ReLU activation functions.
He Initialization and Variance Scaling Initialization is the same thing. In fact, both He and Xavier Initialization are so similar to each other that they can be considered variants of the same idea. Common wisdom in the deep learning world is that sigmoid activation is bad and shouldn't be used. So you don't need to worry about what initialization to use in that case. :)
Also, for best performance in deep networks you need to investigate in batch normalization -- it is not enough to use good initializations.