4
$\begingroup$

I got a problem where batch normalization before the first non-linear activation is a bad idea. Imagine that a neural network has to know the original value of some inputs to get a job done. Inclusion of batch normalization before the first activation will scale these input values according to other inputs of the sample and the neural network will be confused by that transform.

Imagine next that I don't do the batch normalization before the first activation, but I instead normalize in the inputs either by z-transform or mini-max transform. Then the NN get the input information in a correct way. Now I introduce the second hidden layer with non-linear activations.

Is the problem described earlier going to persist if I put batch normalization before the second activations?

  • My question is exactly about a fully-connected NN.
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First, from your description I suppose that you misunderstand BN. Basically normalization is done along the batch axis, not within any dimensions of a sample. (Actually it is possible but this is a special case for convolution layers).

Imagine that a neural network has to know the original value of some inputs to get a job done.

It is hard to imagine, as NN does not have any notion of unit (at least for inputs). It was shown that inputs normalized to mean 0 and variance 1 ease learning.

Anyway, after first activation all the original values are gone, e.g. if activation is tanh, values will be in (-1, 1).

Improve this answer
$\endgroup$
6
  • $\begingroup$ thank you. "First, from your description I suppose that you misunderstand BN". I may be not clear in my wording. BN z-transforms values along outputs of a previous layer, doesn't it? In case of the first hidden layer, these become the network inputs. $\endgroup$
    – Alexey Burnakov
    Feb 9, 2018 at 10:54
  • 2
    $\begingroup$ "Normalization is done along the batch axis, not within any dimensions of a sample" - that's not exactly true. Ioffe and Szegedy experimented with different ways of applying BN and introduced the notion of effective mini-batch - stackoverflow.com/q/38553927/712995 $\endgroup$
    – Maxim
    Feb 9, 2018 at 11:36
  • 1
    $\begingroup$ @AlexeyBurnakov BN transforms along examples in a mini-batch. Values of one example are not mixed. See first part of Maxim's answer to this question stackoverflow.com/q/38553927/712995 $\endgroup$
    – hans
    Feb 9, 2018 at 12:10
  • $\begingroup$ @hans, thank you. I may be confused by the difference between the convolutional layer batch meaning and that of a fully connected layer. Let me check the original paper before I go to the answer. My question was exactly about a fully-connected NN. $\endgroup$
    – Alexey Burnakov
    Feb 9, 2018 at 12:17
  • 1
    $\begingroup$ @AlexeyBurnakov True, you are right. Small batch sizes introduce noise, which may be unwanted but can also work as a kind of regularization. Best would be to check a few batch sizes during hyper-parameter optimization. $\endgroup$
    – hans
    Feb 9, 2018 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.