12
$\begingroup$

What are the pros and cons of weight normalization vs batch normalization in general and for convolutional neural networks specifically?

$\endgroup$
14
$\begingroup$

Batch Norm:

  • (+) Stable if the batch size is large
  • (+) Robust (in train) to the scale & shift of input data
  • (+) Robust to the scale of weight vector
  • (+) Scale of update decreases while training
  • (-) Not good for online learning
  • (-) Not good for RNN, LSTM
  • (-) Different calculation between train and test

Weight Norm:

  • (+) Smaller calculation cost on CNN
  • (+) Well-considered about weight initialization
  • (+) Implementation is easy
  • (+) Robust to the scale of weight vector
  • (-) Compared with the others, might be unstable on training
  • (-) High dependence to input data

Layer Norm:

  • (+) Effective to small mini batch RNN
  • (+) Robust to the scale of input
  • (+) Robust to the scale and shift of weight matrix
  • (+) Scale of update decreases while training
  • (-) Might be not good for CNN (Batch Norm is better in some cases)
$\endgroup$
3
  • 2
    $\begingroup$ Can you flesh out why Batch Norm might be better than Layer Norm for CNN? $\endgroup$ – Eric Auld Mar 12 '18 at 19:00
  • 4
    $\begingroup$ Also, are there any sources for these principles? $\endgroup$ – Eric Auld Mar 24 '18 at 16:36
  • $\begingroup$ Looks like the list is the same as this reddit thread, and they link to a slide deck for Layer Normalization at NIPS that presumably gives a table laying out these differences. However, it's in Japanese. $\endgroup$ – adamconkey Nov 14 '19 at 21:46
7
$\begingroup$

https://arxiv.org/abs/1709.08145

'... a detailed comparison of BN and WN algorithms using ResNet-50 network trained on ImageNet. We found that although WN achieves better training accuracy, the final test accuracy is significantly lower (≈6%) than that of BN. This result demonstrates the surprising strength of the BN regularization effect which we were unable to compensate for using standard regularization techniques like dropout and weight decay. We also found that training of deep networks with WN algorithms is significantly less stable compared to BN, limiting their practical applications.'

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.