I am creating a neural network using batchnorm as a regularization method to enable deep models and prevent overfitting.
I understand that batchnorming supresses the internal covariance shift introduced by randomly chosen minibatches, so it generally makes sense to add it to every layer in the model. However, the batchnorm also introduces noise into the training data, which has a particularly heavy effect on model output in the last few layers of the model.
How exactly should I handle the batchnorm at the end of my model? It seems intuitive that the last layer should not be batchnormed, as this would reduce the degrees of freedom for the output of the model and shift the outputs uncontrollably. But what about the penultimate layer and those before? Would it make sense to "lighten" the effects of the minibatch for example by reducing its impact gradually? Or is it better to just completely leave it out after some layer?