When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want to measure the distance/error between input and output or in general input and label of the network. On the other hand, I also always thought, that binary cross entropy is only used, when we try to predict probabilities and the ground truth label entries are actual probabilities.
Now when working with the mnist dataset loaded via tensorflow like so:
from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
Each entry is a float32 and ranges between 0 and 1.
The tensorflow tutorial for autoencoder(https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb) uses R2-loss/MSE-loss for measuring the reconstruction loss.
Where as the tensorflow tutorial for variational autoencoder (https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/variational_autoencoder.ipynb) uses binary cross entropy for measuring the reconstruction loss.
Can some please tell me WHY, based on the same dataset with same values (they are all numerical values which in effect represent pixel values) they use R2-loss/MSE-loss for the autoencoder and Binary-Cross-Entropy loss for the variational autoencoder.
I think it is needless to say, that both loss functions are applied on sigmoid outputs.