Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

Question

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want to measure the distance/error between input and output or in general input and label of the network. On the other hand, I also always thought, that binary cross entropy is only used, when we try to predict probabilities and the ground truth label entries are actual probabilities.

Now when working with the mnist dataset loaded via tensorflow like so:

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

Each entry is a float32 and ranges between 0 and 1.

The tensorflow tutorial for autoencoder uses R2-loss/MSE-loss for measuring the reconstruction loss.

Where as the tensorflow tutorial for variational autoencoder uses binary cross-entropy for measuring the reconstruction loss.

Can some please tell me WHY, based on the same dataset with same values (they are all numerical values which in effect represent pixel values) they use R2-loss/MSE-loss for the autoencoder and Binary-Cross-Entropy loss for the variational autoencoder.

I think it is needless to say, that both loss functions are applied on sigmoid outputs.

Answer 1

I don't believe there's some kind of deep, meaningful rationale at play here - it's a showcase example running on MNIST, it's pretty error-tolerant.

---

Optimizing for MSE means your generated output intensities are symmetrically close to the input intensities. A higher-than-training intensity is penalized by the same amount as an equally valued lower intensity.

---

Cross-entropy loss is assymetrical.

If your true intensity is high, e.g. 0.8, generating a pixel with the intensity of 0.9 is penalized more than generating a pixel with intensity of 0.7.

Conversely if it's low, e.g. 0.3, predicting an intensity of 0.4 is penalized less than a predicted intensity of 0.2.

You might have guessed by now - cross-entropy loss is biased towards 0.5 whenever the ground truth is not binary. For a ground truth of 0.5, the per-pixel zero-normalized loss is equal to 2*MSE.

This is quite obviously wrong! The end result is that you're training the network to always generate images that are blurrier than the inputs. You're actively penalizing any result that would enhance the output sharpness more than those that make it worse!

---

MSE is not immune to the this behavior either, but at least it's just unbiased and not biased in the completely wrong direction.

However, before you run off to write a loss function with the opposite bias - just keep in mind pushing outputs away from 0.5 will in turn mean the decoded images will have very hard, pixellized edges.

That is - or at least I very strongly suspect is - why adversarial methods yield better results - the adversarial component is essentially a trainable, 'smart' loss function for the (possibly variational) autoencoder.

Answer 2

This discussion suggests that binary cross entropy is used in VAE case mainly for better optimization behavior. Another reason it works well is that MNIST dataset roughly follows multivariate Bernoulli distribution - the pixel values are close to either zero or one and binarization does not change it much. For more in-depth explanation of this, see Using a Bernoulli VAE on real-valued observations.

Answer 3

It depends on how you assume the model for the likelihood. In other words, in variational autoencoders you seek to minimize the ELBO (empirical lower bound), which contains $KL(q||p)$ which is managed by the encoder and a second term known as the reconstruction error $E_{q}[log p(x|z)]$ managed by the decoder and requires sampling, here is where the choise of the model of $p(x|z)$ comes into play. If you assume it follows a normal distribution you will end up with a MSE minimization since $p(x|z)$ can be reformulated as $p(x|\hat{x}) \sim \mathcal{N}(\hat{x},\sigma)$, if you assume a multinoully distribution you will use cross entropy.

Just a side note taken from Goodfellow's book: Many authors use the term “cross-entropy” to identify specifically the negative log-likelihood of a Bernoulli or softmax distribution, but that is a misnomer. Any loss consisting of a negative log-likelihood is a crossentropy between the empirical distribution defined by the training set and the probability distribution defined by model. For example, mean squared error is the cross-entropy between the empirical distribution and a Gaussian model.