I have a dynamic range of 256x256 matrices. I want to have a CNN based binary classifier. The matrices are images with a very wide range of intensities (10 orders of magnitude).

I am afraid to use Mean Normalization or Min-Max Scaling, because it will introduce negative values into the feature space:

$x- \bar{x} \over \sigma$ or $\frac{x - \bar{x}}{\text{max}(x)-\text{min}(x)}$

Assume I am using z-score normalization. All the negative values will go to zero after the first activation layer. Isn't it true? So basically I am loosing all the information in the regions at which the signal is bellow the average.

  • $\begingroup$ Min-max scaling scales to [0, 1] (but the usual formula is that you subtract min instead of mean), so does not introduce negative values. Why negative values are a problem? $\endgroup$
    – Tim
    Oct 16, 2018 at 7:14
  • $\begingroup$ @Tim, but it's sensitive to outliers. My intuitions says that all the CNN filters will be sensitive to the sign. For example assume a standard dog/cat classification problem. Now, we will extend the inputs by adding also the $-1\cdot$ images. Will the training converge into something stable? $\endgroup$
    – 0x90
    Oct 16, 2018 at 7:19
  • $\begingroup$ The second formula does not scale to [0,1], it would have to be $[x-\mathrm{min}(x)] / [\mathrm{max}(x) - \mathrm{min}(x)]$. Is that what you actually had in mind? $\endgroup$ Oct 16, 2018 at 7:19
  • $\begingroup$ @JanKukacka, yes. because the min could be very small. Although maybe [x−min(x)]/[max(x)−min(x)] is also a good idea. What if I create 3 channel inputs with these 3 normalization? $\endgroup$
    – 0x90
    Oct 16, 2018 at 7:21
  • $\begingroup$ There is no requirement for neural networks that inputs need to be positive... You have weights and bias terms everywhere, so if you multiplied inputs with -1, then to get equivalent weights multiply them by -1, if you shift inputs by constant, adjust bias etc. The role of scaling is to have inputs with same magnitude. $\endgroup$
    – Tim
    Oct 16, 2018 at 7:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.