Why is binary cross entropy (or log loss) used in autoencoders for non-binary data

Question

I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that many people (e.g., the keras tutorial on autoencoders, this guy) use binary cross-entropy as the loss function in this scenario. While the autoencoder works, it produces slightly blurry reconstructions, which, among many reasons, might be because binary cross-entropy for non-binary data penalizes errors towards 0 and 1 more than errors towards 0.5 (as nicely explained here).

For example, give the true value is 0.2, and autoencoder A predicts 0.1 while autoencoder 2 predicts 0.3. The loss for A would be

−(0.2 * log(0.1) + (1−0.2) * log(1−0.2)) = .27752801

while the loss for B would be

−(0.2 * log(0.3) + (1−0.2) * log(1−0.3)) = .228497317

Hence, the B is considered to be a better reconstruction than A; if I got everything correct. But this does not exactly make sense to me as I am not sure why asymmetric is preferred over other symmetric loss functions like MSE.

In this video Hugo Larochelle argues that the minimum will still be at the point of perfect reconstruction, but the loss will never be zero (which makes sense). This is further explained in this excellent answer, which proves why the minimum of binary cross-entropy for non-binary values that are in [0,1] is given when the prediction equals the true value.

So, my question is: Why is binary cross-entropy used for non-binary values in [0,1] and why is the asymmetric loss is acceptable compared to other symmetric loss functions like MSE, MAE, etc.? Does it have a better loss landscape, i.e., is it convex while others are not, or are there other reasons?

Answer 1

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $\rm-loss$ as a function of NN output $x$ and find its derivative:

\begin{align} f(x) &= a \ln x + (1-a) \ln (1-x)\\ f^\prime(x) &= \frac{a-x}{x(1-x)} \end{align}

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$$ f^\prime(\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $$

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

Answer 2

If you think the loss from 0.1 and 0.3 should be equal when the true is 0.2, there is no reason to use the cross entropy. The loss function should reflect what your or your field's common sense.

However, if the true value $p$ is corresponding to a Bernoulli distribution with mean $p$, then, cross entropy loss between $p$ and $q$ is equal to the KL divergence between $\operatorname{Ber}(p)$ and $\operatorname{Ber}(q)$ which is one of the most natural and optimal loss in some senses.

In general, every strongly convex loss behaves similarly to the $l_2$ loss around the true value. So the sensitivity of the choice of loss will vanish as your prediction getting accurate in any loss.

Answer 3

A change of 0.1 in either direction introduces a symmetric additive effect, but an asymmetric multiplicative effect.

This means that while both A and B are the same shift from the true mean, the true value is twice A but 2/3 of B. Inversely A if half the true value, B is 1.5 times it. I. E. Their multiplicative distances are different.

One would use a symmetric function when one is evaluating something expected to be symmetric, an asymmetric one for asymmetric situations. Note that logs are used because they allow us to handle multiplicative processes in a more additive way.

Answer 4

Under the Bernoulli distribution parameterized by say $p = 0.3$ by the output of the autoencoder, the probability of drawing $x = 0.2$ is zero (and is zero for all $0 < x < 1$). This indeed makes the Bernoulli distribution a bad choice for non-binary data.

However, a slightly different view of the input can resurrect the Bernoulli distribution. Let's assume instead that $x = 0.2$ is a sample from some measuring device, and this $x = 0.2$ reading might be best described as itself being a parameter of a probability distribution, such as a normal or Bernoulli distribution. Let's go with the latter and say that $x = 0.2$ represents a Bernoulli process with parameter $p' = x = 0.2$. Thus, there is some underlying binary sensor or event which is $0$ with probability $0.2$ and $1$ with probability $0.8$. The output of our autoencoder is a Bernoulli distribution with say $p = 0.3$. It does make sense to ask: what is the expected result of drawing $0$ or $1$ readings from the real Bernoulli process (with parameter $p'=0.2$) and then calculating its likelihood value according to the autoencoder's Bernoulli distribution (with parameter $p = 0.3$). This expected likelihood is $p'p + (1-p')(1-p) = (0.2)(0.3) + (0.8)(0.7)$. We can also ask what the expected log-likelihood is, and that is $p'\log(p) + (1-p')\log(1-p)$. When we replace the symbol $p'$ with the usual symbol, $y$, we get the usual expression $y\log(p) + (1-y)\log(1-p)$.

By interpreting the input differently (as a distribution parameter), the cross-entropy loss does make sense as the negative of the expected log-likelihood, where the expectation is over the "input" distribution, and likelihood is calculated against our "output" distribution.