Is BatchNorm regularization technique? Why?
BatchNorm works by standardizing the outputs of hidden units across an entire batch. The standardization process consists of multiplication and addition. Compare this to another regularization technique such as injecting noise into the outputs (or inputs) of hidden units; the noise can be injected additively or multiplicatively. So you can, in a way, think of BatchNorm as a injecting the 'correct noise' needed to standardise hidden unit outputs across a batch, and although it won't be as strong of a regularizing effect as actual uniform/gaussian random noise, it still has a minor regularizing effect on top of the benefit of speeding up learning.
Should we use BatchNorm only during training process? Why?
BatchNorm is used during training to standardise hidden layer outputs, but during evaluation the parameters that the BatchNorm layer has learnt (the mean and standard deviation) are frozen and are used as is, just like all other weights in a network. The effects of BatchNorm can also be 'folded in' to network weights which achieves the same effect but with one less step.
Can we use Dropout and BatchNorm simultaneously? If we can, in what order?
Definitely! Although there is a lot of debate as to which order the layers should go. Older literature claims
Dropout -> BatchNorm is better while newer literature claims that it doesn't matter or that
BatchNorm -> Dropout is superior. My recommendation is try both; every network is different and what works for some might not work for others. Personally I've found
BatchNorm -> Dropout to work well for my use cases.