3
$\begingroup$

Some of the more modern neural network architectures (Densenets, for example) use pre-activation batch normalization i.e batch normalization -> activation -> Convolution rather than the usual Convolution -> batch normalization -> activation.

The resulting network has the same number of parameters but most of the papers report that pre-activation in these networks is better. I am struggling to figure out why this is the case. Is there any intuitive or principled reasoning behind it?

$\endgroup$
2
  • $\begingroup$ Can you share citations of articles that find pre-activation normalized networks are better than post-activation normalized networks? I would be interested to read about them, and I wonder if the authors make passing mention of this phenomenon, perhaps in an appendix or other materials. $\endgroup$
    – Sycorax
    Apr 11, 2019 at 15:04
  • $\begingroup$ @Syrocax: I found this nice blog post discussing it: learningstracker.wordpress.com/2017/01/04/… $\endgroup$
    – Luca
    Apr 11, 2019 at 15:15

1 Answer 1

1
$\begingroup$

A convolutional unit of the form BN-ReLU-conv is convenient since it applies the batchnorm immediately after summation with the skip connection.

The original form (conv-BN-ReLu) allows the summation to spoil the normalization contributed by batchnorm. Additionally, the original form places a non-linear activation after the summation and before the next skip connection. This impedes the gradient flow towards earlier layers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.