In Tensorflow's implementation of ResNet, I find they use variance scaling initializer, I also find xavier initializer is popular. I don't have too much experience on this, which is better in practice?
Xavier initialization, originally proposed by Xavier Glorot and Yoshua Bengio in "Understanding the difficulty of training deep feedforward neural networks", is the weights initialization technique that tries to make the variance of the outputs of a layer to be equal to the variance of its inputs. This idea turned out to be very useful in practice. Naturally, this initialization depends on the layer activation function. And in their paper, Glorot and Bengio considered logistic sigmoid activation function, which was the default choice at that moment.
Later on, the sigmoid activation was surpassed by ReLu, because it allowed to solve vanishing / exploding gradients problem. Consequently, there appeared a new initialization technique, which applied the same idea (balancing of the variance of the activation) to this new activation function. It was proposed by Kaiming He at al in "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification", and now it often referred to as He initialization.
In tensorflow, He initialization is implemented in
variance_scaling_initializer() function (which is, in fact, a more general initializer, but by default performs He initialization), while Xavier initializer is logically
In summary, the main difference for machine learning practitioners is the following:
- He initialization works better for layers with ReLu activation.
- Xavier initialization works better for layers with sigmoid activation.
Variance scaling is just a generalization of Xavier: http://tflearn.org/initializations/. They both operate on the principle that the scale of the gradients should be similar throughout all layers. Xavier is probably safer to use since it's withstood the experimental test of time; trying to pick your own parameters for variance scaling might inhibit training or cause your network to not earn at all.