I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that many people (e.g., the keras tutorial on autoencoders, this guy) use binary cross-entropy as the loss function in this scenario. While the autoencoder works, it produces slightly blurry reconstructions, which, among many reasons, might be because binary cross-entropy for non-binary data penalizes errors towards 0 and 1 more than errors towards 0.5 (as nicely explained here).

For example, give the true value is 0.2, and autoencoder A predicts 0.1 while autoencoder 2 predicts 0.3. The loss for A would be

−(0.2 * log(0.1) + (1−0.2) * log(1−0.2)) = .27752801

while the loss for B would be

−(0.2 * log(0.3) + (1−0.2) * log(1−0.3)) = .228497317

Hence, the B is considered to be a better reconstruction than A; if I got everything correct. But this does not exactly make sense to me as I am not sure why asymmetric is preferred over other symmetric loss functions like MSE.

In this video Hugo Larochelle argues that the minimum will still be at the point of perfect reconstruction, but the loss will never be zero (which makes sense). This is further explained in this excellent answer, which proves why the minimum of binary cross-entropy for non-binary values that are in [0,1] is given when the prediction equals the true value.

So, my question is: Why is binary cross-entropy used for non-binary values in [0,1] and why is the asymmetric loss is acceptable compared to other symmetric loss functions like MSE, MAE, etc.? Does it have a better loss landscape, i.e., is it convex while others are not, or are there other reasons?

  • $\begingroup$ Funny, I pasted your calculations into python and I got 0.639 and 0.526 respectively. Probably you used log based on 10 and I used natural logarithm. $\endgroup$ – hans Mar 7 '19 at 17:49
  • $\begingroup$ Indeed, I used log10, but the base doesn't matter. What matters is the asymmetry and what its benefits are. Maybe log loss and mse are both equally useful. I was just wondering as it appeared to me that ~50% use log loss for image autoencoders and the other ~50% use mse. $\endgroup$ – Flek Mar 7 '19 at 19:00
  • $\begingroup$ Sure, I was just joking :-) I understand your question. For binary data answer is obvious - gradient behaves as constant when prediction is very close to true value (for log), while for mse it is linear, i.e. it becomes small. $\endgroup$ – hans Mar 7 '19 at 22:02
  • $\begingroup$ There is a typo in calculations, but yes loss is asymmetric: >>> -0.2 * np.log(0.1) - (1.0-0.2) * np.log(1.0-0.1) 0.5448054311250702 >>> -0.2 * np.log(0.3) - (1.0-0.2) * np.log(1.0-0.3) 0.5261345160161732 $\endgroup$ – mrgloom Jul 30 at 16:13

Your question inspired me to have a look on loss function from point of view of mathematical analysis. This is a disclaimer - my background is in physics, not in statistics.

Let's rewrite $-loss$ as a function of NN output $x$ and find its derivative:

$ f(x) = a \ln x + (1-a) \ln (1-x) $

$ f'(x) = \frac{a-x}{x(1-x)} $

where $a$ is the target value. Now we put $x = a + \delta$ and assuming that $\delta$ is small we can neglect terms with $\delta^2$ for clarity:

$ f'(\delta) = \frac{\delta}{a(a-1) + \delta(2a-1)} $

This equation let us get some intuition how loss behaves. When target value $a$ is (close to) zero or one, derivative is constant $-1$ or $+1$. For $a$ around 0.5 the derivative is linear in $\delta$.

In other words, during backpropagation this loss cares more about very bright and very dark pixels, but puts less effort on optimizing middle tones.

Regarding assymetry - when NN is far from optimum, it does not matter probably, as you will converge faster or slower. When NN is close to optimum ($\delta$ is small) assymetry disappears.

| cite | improve this answer | |

If you think the loss from 0.1 and 0.3 should be equal when the true is 0.2, there is no reason to use the cross entropy. The loss function should reflect what your or your field's common sense.

However, if the true value $p$ is corresponding to a Bernoulli distribution with mean $p$, then, cross entropy loss between $p$ and $q$ is equal to the KL divergence between $Ber(p)$ and $Ber(q)$ which is one of the most natural and optimal loss in some senses.

In general, every strongly convex loss behaves similarly to the $l_2$ loss around the true value. So the sensitivity of the choice of loss will vanish as your prediction getting accurate in any loss.

| cite | improve this answer | |
  • $\begingroup$ I always thought that you use log because it does not behave like $l_2$ close to zero. $\endgroup$ – hans Mar 6 '19 at 10:20
  • $\begingroup$ Yes, definitely, if the mean parameter is too close to 0 or 1, cross entropy and l2 loss behave differently. When I said strongly convex loss behaves similarly to l2, it meant when the underlying mean parameters are not too close to the boundary. $\endgroup$ – JaeHyeok Shin Mar 7 '19 at 17:34

A change of 0.1 in either direction introduces a symmetric additive effect, but an asymmetric multiplicative effect.

This means that while both A and B are the same shift from the true mean, the true value is twice A but 2/3 of B. Inversely A if half the true value, B is 1.5 times it. I. E. Their multiplicative distances are different.

One would use a symmetric function when one is evaluating something expected to be symmetric, an asymmetric one for asymmetric situations. Note that logs are used because they allow us to handle multiplicative processes in a more additive way.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.