Softmax weights initialization I am a new to deep learning and neural networks, and I need to know if there is a good weights initialization method to use if the activation function is Softmax like Tanh, ReLU and Sigmoid. Related answer. 
 A: For:


*

*ReLU and variants like PReLU, RReLU and ELU: use He initialization (uniform or normal)

*SELU: use LeCun initialization (normal) (see this paper)

*Default (including Sigmoid, Tanh, Softmax, or no activation): use Xavier initialization (uniform or normal), also called Glorot initialization. This is the default in Keras and most other deep learning libraries.


When initializing the weights with a normal distribution, all these methods use mean 0 and variance σ²=scale/fan_avg or σ²=scale/fan_in. The fan_in is the layer's number of inputs, the fan_out is the layer's number of outputs (=number of neurons), fan_avg is the average between the two =½(fan_in+fan_out). Specifically:


*

*Xavier: σ²=1/fan_avg

*He: σ²=2/fan_in

*LeCun: σ²=1/fan_in


When initializing the weights with a uniform distribution, all these methods just use the range [-limit, limit] where limit = sqrt(3 * σ²).
If you have consecutive ReLU layers with very different sizes, you may prefer using fan_avg rather than fan_in. In Keras, you can use something like this:
init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg', distribution='normal')
layer = Dense(10, activation="relu", kernel_initializer=init)

A: He or Xavier Initialization is usually what is recommended. See for
example
this
article:

I recommend simply remembering:
Use Xavier Initialization in the fully-connected layers of your network. (Or layers that use
  softmax/tanh activation functions)
Use Variance Scaling Initialization in the intermediate layer of your network that use
  ReLU activation functions.

He Initialization and Variance Scaling Initialization is the same
thing. In fact, both He and Xavier Initialization are so similar to
each other that they can be considered variants of the same
idea. Common wisdom in the deep learning world is that sigmoid
activation is bad and shouldn't be used. So you don't need to worry
about what initialization to use in that case. :)
Also, for best performance in deep networks you need to investigate in
batch normalization -- it is not enough to use good initializations.
