Neural Network Input Normalization

Question

As I know it is recommended to normalize the input to a neural network ($\mu=0,\sigma=1$). Assuming I use a CNN that uses RGB-images as input, how do i have to normalize?

• Over all the data of the complete batch (at once)?
• Each image independent of each other?
• For each $x$/$y$ pair (=pixel) over the batch?
  • For each color channel or all at once?

Thank you very much

-edit-

Or is it sufficient to use a BatchNormalization-Layer as the first layer in the neural network?

Answer 1

Usually, for RGB images you just subtract half the range and divide by the full range: $$x:=\frac{x-\frac{range}{2}}{range}$$ where $range=255$ in this case since each pixel is a value from 0 to 255. This will not result in having zero mean and unit variance but for most applications it is close enough and works well.

If this is not working for you, the correct way to scale the data to have zero mean and unit variance is to take each channel individually; calculate the mean and variance of all the pixels in all the images in that channel; then scale each channel individually as follows: $$x_R:=\frac{x_R-\mu_R}{\sigma_R} \;\; x_G:=\frac{x_G-\mu_G}{\sigma_G} \;\; x_B:=\frac{x_B-\mu_B}{\sigma_B}$$

Each pixel in a channel is the same feature, just in a different position in space, this is especially true when using a CNN.

Note: If $\sigma$ is too small for one or more of the channels you can cap it at minimum value to avoid numerical stability issues e.g. $\sigma= max(\sigma, 0.0001)$

Answer 2

I have experienced that input image normalization doesn't improve the classification accuracy more than 10% from batch normalization after the first convolution layer. My input image was grey scale looking like metal surface microscopic photo. My neural network model was similar to AlexNet.