Input data normalization

Question

I saw it sometimes that the input data for Neural Networks was scaled to $[0, 1]$ (or $[-1, 1]$), but also sometimes that it was normalized.

E.g. for a greyscale image the normalization was done using the distribution of the different pixel values of the specific input image. But this has the effect that different images (shifted images) result in the same input for the neural network? Isn't this bad?

Or does it make sense to just insert a BatchNormalization-layer as the initial layer of a Neural Network? If not: Why not?

Or should in general no normalization be done, but only scaling the data to a new specific range?

Answer 1

Preprocessing makes the learning of the neural network much easier. Consider the classical example where we want to classify if a given picture contains a dog. In this case, no information is included in the fact what exposure time was chosen when taken the photo.

If you feed the unprocessed data into your learning algorithm, it has to learn that a bright picture of a dog might show the same as a dark picture of a dog. When you preprocess the data, the algorithm will get a nearly identical picture in both cases, so it is not necessary to learn this fact. Thus it is faster and more robust.

But it very much depends on what you want to find. If you want to differentiate pictures with dogs at night from pictures with dogs at sunshine, you should not do this preprocessing step (maybe another one), because there is actually information encoded in the brightness of the picture that helps the neural network to make the decision.

Using Batch Normalization is not as robust as doing a good preprocessing, because you can put in a lot of application logic into your preprocessing. However, if you really have no application knowledge, Batch Normalization can still speed up your learning (sometimes a lot), but I would see it as supplement, not as replacement.