# Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want to measure the distance/error between input and output or in general input and label of the network. On the other hand, I also always thought, that binary cross entropy is only used, when we try to predict probabilities and the ground truth label entries are actual probabilities.

Now when working with the mnist dataset loaded via tensorflow like so:

from tensorflow.examples.tutorials.mnist import input_data


Each entry is a float32 and ranges between 0 and 1.

The tensorflow tutorial for autoencoder(https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb) uses R2-loss/MSE-loss for measuring the reconstruction loss.

Where as the tensorflow tutorial for variational autoencoder (https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/variational_autoencoder.ipynb) uses binary cross entropy for measuring the reconstruction loss.

Can some please tell me WHY, based on the same dataset with same values (they are all numerical values which in effect represent pixel values) they use R2-loss/MSE-loss for the autoencoder and Binary-Cross-Entropy loss for the variational autoencoder.

I think it is needless to say, that both loss functions are applied on sigmoid outputs.

I don't believe there's some kind of deep, meaningful rationale at play here - it's a showcase example running on MNIST, it's pretty error-tolerant.

Optimizing for MSE means your generated output intensities are symmetrically close to the input intensities. A higher-than-training intensity is penalized by the same amount as an equally valued lower intensity.

Cross-entropy loss is assymetrical.

If your true intensity is high, e.g. 0.8, generating a pixel with the intensity of 0.9 is penalized more than generating a pixel with intensity of 0.7.

Conversely if it's low, e.g. 0.3, predicting an intensity of 0.4 is penalized less than a predicted intensity of 0.2.

You might have guessed by now - cross-entropy loss is biased towards 0.5 whenever the ground truth is not binary. For a ground truth of 0.5, the per-pixel zero-normalized loss is equal to 2*MSE.

This is quite obviously wrong! The end result is that you're training the network to always generate images that are blurrier than the inputs. You're actively penalizing any result that would enhance the output sharpness more than those that make it worse!

MSE is not immune to the this behavior either, but at least it's just unbiased and not biased in the completely wrong direction.

However, before you run off to write a loss function with the opposite bias - just keep in mind pushing outputs away from 0.5 will in turn mean the decoded images will have very hard, pixellized edges.

That is - or at least I very strongly suspect is - why adversarial methods yield better results - the adversarial component is essentially a trainable, 'smart' loss function for the (possibly variational) autoencoder.

• So first of all, you are saying that in the toy example with MNIST dataset, the loss function does not make a big difference as it is trainable/converging for both approaches? What about the reconstruction loss function for VAEs with bilevel datasamples (i.e. each point can be either 0 or 1), what loss function would one use here? And would it be feasible to always use the MSE loss? – SolingerMUC Jun 10 '18 at 14:41
• Mostly, I'm saying it doesn't make a big difference because it's not a model intended for real-life deployment - it's just a very rough proof of concept, you got some digits to show, close enough for government work. If your data is a binary label, on/off, binary cross-entropy will be just about perfect - that's what it was designed for. MSE is... serviceable, I suppose - mostly if your activation function is constrained to a 0->1 range AND your target is strictly binary; any other case and it starts to tear at the seams. – jkm Jun 10 '18 at 15:20
• Regarding your actual answer, what do you mean by "cross-entropy loss is biased towards 0.5 whenever the ground truth is not binary" ? And I also did not get what you meant with "a loss function with the opposite bias" ? Thanks already for the great discussion :)! – SolingerMUC Jun 10 '18 at 18:28
• Likewise, glad to hear you're enjoying it! It's basically just restating the same thing I said a moment earlier - the loss is higher for outputs further from 0.5 than the inputs. So for ground truth of 0.3, outputting 0.2 is treated as worse than outputting 0.4, and likewise for 0.7, outputting 0.8 is discouraged more than a 0.6. – jkm Jun 11 '18 at 17:55
• The alternative loss - that would be a function that also has a minimum when p(x) == q(x) and increases assymetrically if they are different, but favors 'overshooting' the probability away from 0.5 in either direction, something along the lines of KLD from a bimodal Gaussian I suppose. – jkm Jun 11 '18 at 18:07

This discussion suggests that binary cross entropy is used in VAE case mainly for better optimization behavior. Another reason it works well is that MNIST dataset roughly follows multivariate Bernoulli distribution - the pixel values are close to either zero or one and binarization does not change it much. For more in-depth explanation of this, see Using a Bernoulli VAE on real-valued observations.