I got a problem where batch normalization before the first non-linear activation is a bad idea. Imagine that a neural network has to know the original value of some inputs to get a job done. Inclusion of batch normalization before the first activation will scale these input values according to other inputs of the sample and the neural network will be confused by that transform.
Imagine next that I don't do the batch normalization before the first activation, but I instead normalize in the inputs either by z-transform or mini-max transform. Then the NN get the input information in a correct way. Now I introduce the second hidden layer with non-linear activations.
Is the problem described earlier going to persist if I put batch normalization before the second activations?
- My question is exactly about a fully-connected NN.