# What are the best practices to normalize inputs before training a CNN?

I have a dynamic range of 256x256 matrices. I want to have a CNN based binary classifier. The matrices are images with a very wide range of intensities (10 orders of magnitude).

I am afraid to use Mean Normalization or Min-Max Scaling, because it will introduce negative values into the feature space:

$$x- \bar{x} \over \sigma$$ or $$\frac{x - \bar{x}}{\text{max}(x)-\text{min}(x)}$$

Assume I am using z-score normalization. All the negative values will go to zero after the first activation layer. Isn't it true? So basically I am loosing all the information in the regions at which the signal is bellow the average.

• Min-max scaling scales to [0, 1] (but the usual formula is that you subtract min instead of mean), so does not introduce negative values. Why negative values are a problem?
– Tim
Oct 16, 2018 at 7:14
• @Tim, but it's sensitive to outliers. My intuitions says that all the CNN filters will be sensitive to the sign. For example assume a standard dog/cat classification problem. Now, we will extend the inputs by adding also the $-1\cdot$ images. Will the training converge into something stable?
– 0x90
Oct 16, 2018 at 7:19
• The second formula does not scale to [0,1], it would have to be $[x-\mathrm{min}(x)] / [\mathrm{max}(x) - \mathrm{min}(x)]$. Is that what you actually had in mind? Oct 16, 2018 at 7:19
• @JanKukacka, yes. because the min could be very small. Although maybe [x−min(x)]/[max(x)−min(x)] is also a good idea. What if I create 3 channel inputs with these 3 normalization?
– 0x90
Oct 16, 2018 at 7:21
• There is no requirement for neural networks that inputs need to be positive... You have weights and bias terms everywhere, so if you multiplied inputs with -1, then to get equivalent weights multiply them by -1, if you shift inputs by constant, adjust bias etc. The role of scaling is to have inputs with same magnitude.
– Tim
Oct 16, 2018 at 7:40