When people use autoencoders, they usually normalize the data such that the values are normalized to the range [0,1]. Why is that? Why not use zero-mean unit variance normalization for example? I read on a Quora answer that this range gives you more choice of loss functions, but I don't really understand why. Any ideas?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Most likely the context here is autoencoders on images. With RGB, you have 256 colors for each channel, hence the input and output are bounded for each pixel, and therefore equivalent to $[0,1]$ after normalization. $\endgroup$– Alex R.Sep 27, 2017 at 19:32
-
$\begingroup$ In that paper on biorxiv.org/content/early/2017/08/11/174474, they normalize the RNA-Seq levels to [0,1] as well while they are applying VAE on the gene expression data -- so, no images there. $\endgroup$– user5054Sep 27, 2017 at 19:37
-
$\begingroup$ Check out this detailed answer for normalization and dropout for autoencoder : datascience.stackexchange.com/questions/32901/… $\endgroup$– Aaditya UraDec 27, 2019 at 13:50
-
$\begingroup$ Isn't there a rule of thumb encourages using 0-1 encoding for categorical variables? $\endgroup$– Oskar_UNov 1, 2020 at 0:57
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
In general, the exactly normalization of data isn't super important in neural networks as long as the inputs are at some reasonable scale. As Alex mentioned, with images, normalization to 0 and 1 happens to be very convenient.
The fact that normalization doesn't matter much is only made stronger by use of batch-normalization, which is a function/layer frequently used in neural networks which renormalizes the activations halfway through the network to zero mean and unit variance. And the authors of the paper you linked did use batch normalization, which means however the data was normalized before, it was renormalized a bunch of times inside the network anyway.
Furthermore, reading their code, which is on github, they actually did preprocess the data two ways -- with zero mean unit variance normalization, and also with min 0 max 1 normalization. They didn't explain why they chose one preprocessed dataset over the other, but I suspect they either just arbitrarily to use min 0 max 1 normalization, or some preliminary hyperparameter searches showed that one worked better than the other for whatever reason.
-
$\begingroup$ Thanks! Could you explain why batch-normalization is needed after initial normalization and why zero-mean unit variance normalization is used for that? I am quite new to the concept of neural networks, so, sorry if this sounds like a dumb question. $\endgroup$– user5054Sep 28, 2017 at 19:18
-
$\begingroup$ Even if the data is initially normalized, after it is passed through many linear transformations and nonlinear functions inside the neural network, the resulting values are no longer normalized, which may cause difficult in the deeper layers. Batch normalization is a layer which can be placed at any point inside the network which takes values at that layer and normalizes them. It is commonly applied after every layer. I suspect that zero-mean unit variance is used in BN instead of min-max normalization because it is more differentiable. $\endgroup$– shimaoSep 28, 2017 at 19:22