When people use autoencoders, they usually normalize the data such that the values are normalized to the range [0,1]. Why is that? Why not use zero-mean unit variance normalization for example? I read on a Quora answer that this range gives you more choice of loss functions, but I don't really understand why. Any ideas?

  • $\begingroup$ Most likely the context here is autoencoders on images. With RGB, you have 256 colors for each channel, hence the input and output are bounded for each pixel, and therefore equivalent to $[0,1]$ after normalization. $\endgroup$ – Alex R. Sep 27 '17 at 19:32
  • $\begingroup$ In that paper on biorxiv.org/content/early/2017/08/11/174474, they normalize the RNA-Seq levels to [0,1] as well while they are applying VAE on the gene expression data -- so, no images there. $\endgroup$ – user5054 Sep 27 '17 at 19:37
  • $\begingroup$ Check out this detailed answer for normalization and dropout for autoencoder : datascience.stackexchange.com/questions/32901/… $\endgroup$ – Aaditya Ura Dec 27 '19 at 13:50
  • $\begingroup$ Isn't there a rule of thumb encourages using 0-1 encoding for categorical variables? $\endgroup$ – Oskar_U Nov 1 '20 at 0:57

In general, the exactly normalization of data isn't super important in neural networks as long as the inputs are at some reasonable scale. As Alex mentioned, with images, normalization to 0 and 1 happens to be very convenient.

The fact that normalization doesn't matter much is only made stronger by use of batch-normalization, which is a function/layer frequently used in neural networks which renormalizes the activations halfway through the network to zero mean and unit variance. And the authors of the paper you linked did use batch normalization, which means however the data was normalized before, it was renormalized a bunch of times inside the network anyway.

Furthermore, reading their code, which is on github, they actually did preprocess the data two ways -- with zero mean unit variance normalization, and also with min 0 max 1 normalization. They didn't explain why they chose one preprocessed dataset over the other, but I suspect they either just arbitrarily to use min 0 max 1 normalization, or some preliminary hyperparameter searches showed that one worked better than the other for whatever reason.

  • $\begingroup$ Thanks! Could you explain why batch-normalization is needed after initial normalization and why zero-mean unit variance normalization is used for that? I am quite new to the concept of neural networks, so, sorry if this sounds like a dumb question. $\endgroup$ – user5054 Sep 28 '17 at 19:18
  • $\begingroup$ Even if the data is initially normalized, after it is passed through many linear transformations and nonlinear functions inside the neural network, the resulting values are no longer normalized, which may cause difficult in the deeper layers. Batch normalization is a layer which can be placed at any point inside the network which takes values at that layer and normalizes them. It is commonly applied after every layer. I suspect that zero-mean unit variance is used in BN instead of min-max normalization because it is more differentiable. $\endgroup$ – shimao Sep 28 '17 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.