2
$\begingroup$

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand,

  1. Dropout is regularization technique, which is using only during training.

  2. BatchNorm is technique, which is using for accelerating training speed, improving accuracy and e.t.c. But I also saw some conflicting opinions about question: is BatchNorm regularization technique?

So, can somebody,please, answer some questions:

  1. Is BatchNorm regularization technique? Why?

  2. Should we use BatchNorm only during training process? Why?

  3. Can we use Dropout and BatchNorm simultaneously? If we can, in what order?

$\endgroup$
3
$\begingroup$

Is BatchNorm regularization technique? Why?

BatchNorm works by standardizing the outputs of hidden units across an entire batch. The standardization process consists of multiplication and addition. Compare this to another regularization technique such as injecting noise into the outputs (or inputs) of hidden units; the noise can be injected additively or multiplicatively. So you can, in a way, think of BatchNorm as a injecting the 'correct noise' needed to standardise hidden unit outputs across a batch, and although it won't be as strong of a regularizing effect as actual uniform/gaussian random noise, it still has a minor regularizing effect on top of the benefit of speeding up learning.


Should we use BatchNorm only during training process? Why?

BatchNorm is used during training to standardise hidden layer outputs, but during evaluation the parameters that the BatchNorm layer has learnt (the mean and standard deviation) are frozen and are used as is, just like all other weights in a network. The effects of BatchNorm can also be 'folded in' to network weights which achieves the same effect but with one less step.


Can we use Dropout and BatchNorm simultaneously? If we can, in what order?

Definitely! Although there is a lot of debate as to which order the layers should go. Older literature claims Dropout -> BatchNorm is better while newer literature claims that it doesn't matter or that BatchNorm -> Dropout is superior. My recommendation is try both; every network is different and what works for some might not work for others. Personally I've found BatchNorm -> Dropout to work well for my use cases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.