27
$\begingroup$

In Tensorflow's implementation of ResNet, I find they use variance scaling initializer, I also find xavier initializer is popular. I don't have too much experience on this, which is better in practice?

$\endgroup$
1
  • 1
    $\begingroup$ For a more detailed explanation of Xavier's initialization, you can visit this link: prateekvjoshi.com/2016/03/29/… It includes proper derivation of Xavier's initialization and intuition behind it. $\endgroup$ Nov 11 '18 at 0:19
47
$\begingroup$

Historical perspective

Xavier initialization, originally proposed by Xavier Glorot and Yoshua Bengio in "Understanding the difficulty of training deep feedforward neural networks", is the weights initialization technique that tries to make the variance of the outputs of a layer to be equal to the variance of its inputs. This idea turned out to be very useful in practice. Naturally, this initialization depends on the layer activation function. And in their paper, Glorot and Bengio considered logistic sigmoid activation function, which was the default choice at that moment.

Later on, the sigmoid activation was surpassed by ReLu, because it allowed to solve vanishing / exploding gradients problem. Consequently, there appeared a new initialization technique, which applied the same idea (balancing of the variance of the activation) to this new activation function. It was proposed by Kaiming He at al in "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification", and now it often referred to as He initialization.

In tensorflow, He initialization is implemented in variance_scaling_initializer() function (which is, in fact, a more general initializer, but by default performs He initialization), while Xavier initializer is logically xavier_initializer().

Summary

In summary, the main difference for machine learning practitioners is the following:

  • He initialization works better for layers with ReLu activation.
  • Xavier initialization works better for layers with sigmoid activation.
$\endgroup$
4
  • 1
    $\begingroup$ in pytorch what initializaiton is used? does it depend on the activation? $\endgroup$ Jul 14 '18 at 1:22
  • 3
    $\begingroup$ Hi, for tanh() activation function, which initialization function should I use. $\endgroup$
    – GoingMyWay
    Oct 22 '18 at 9:29
  • 1
    $\begingroup$ @GoingMyWay, according to this pytorch.org/docs/stable/… , you can use He initialization with gain 5/3. Not sure where that comes from though. $\endgroup$
    – Podgorskiy
    Mar 10 '19 at 18:05
  • 1
    $\begingroup$ tanh is very similar to sigmoid, so you should use Xavier. $\endgroup$ Apr 23 '20 at 13:58
4
$\begingroup$

Variance scaling is just a generalization of Xavier: http://tflearn.org/initializations/. They both operate on the principle that the scale of the gradients should be similar throughout all layers. Xavier is probably safer to use since it's withstood the experimental test of time; trying to pick your own parameters for variance scaling might inhibit training or cause your network to not earn at all.

$\endgroup$
2
  • $\begingroup$ Thanks. What do you mean pick your own parameters for variance? $\endgroup$
    – Hanamichi
    Dec 18 '17 at 21:46
  • $\begingroup$ In the TF implementation, this would be the factor parameter (which determines the ratio of the input variance to the output variance) $\endgroup$
    – liangjy
    Jan 1 '18 at 2:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.