3
$\begingroup$

I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray classification problems where the ground-truth labels (i.e. $y$) are either 0 or 1 and therefore when predictions (i.e. $p$) are correct, in both cases, the loss value would be zero:

$$ BCE(y,p) = -y.log(p) - (1-y).\log{(1-p)} $$

$$ BCE(0,0) = 0, BCE(1,1) = 0 $$

However, binary crossentropy does not have a value of zero when neither of its arguments are both zero or one, which is the case for an autoencoder with ground-truth labels in range $[0,1]$ (i.e. assuming the input data has been normalized in this range). I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. What am I missing here?

$\endgroup$
3
$\begingroup$

I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same.

That's exactly the misconception you have. You think that in order for a loss function to be used in a model like an autoencoder, it must have a value of zero when predictions equal to true labels. That's simply wrong since in most of the machine learning models (including autoencoders) we are trying to minimize a loss/cost function. And we are doing this with the assumption that the loss function we are using when reaches its minimum point, implies that the predictions and true labels are the same. That's the condition for using a function as a loss function in a model trained based on minimzing loss function. Note that the value of loss function at this minimum point may not be zero at all, however we don't care about this as long as it implies in that point predictions and true labels are the same.

Now let's verify this is the case for binary crossentropy: we need to show that when we reach the minimum point of binary crossentropy it implies that $y = p$, i.e. predictions equal to true labels. To find the minimum point, we take the derivative with respect to $p$ and set it equal to zero (note that in the following calculations I have assumed that the $log$ is natural logarithm function to make calculations a little easier):

$$\begin{align}&\frac{\partial BCE(y,p)}{\partial p} = 0\\ &\implies -y.\dfrac{1}{p} - (1-y).\dfrac{-1}{1-p} = 0\\ &\implies -y.(1-p) + (1-y).p = 0\\ &\implies -y + y.p + p - y.p = 0\\ &\implies p - y = 0\\ &\implies y = p \end{align}$$

$\endgroup$
  • $\begingroup$ Could you also point out why one should use binary crossentropy (BC) over e.g. mean-squared-error (MSE) at all? In fact they both have the same minimum so they should be equivalent. However, the outputs (e.g. VAE on MNIST) look different. $\endgroup$ – Tik0 Oct 15 '18 at 8:47
  • $\begingroup$ @Tik0 I don't think VAE is trained using either of MSE or BCE loss functions. Instead, KL-divergence is usually used as the loss function in this specific type of autoencoders. If you have any example of autoencoder trained using MSE and BCE loss and there is a noticable difference between the results obtained, please provide a reference so that I can take a look at it and investigate more. $\endgroup$ – today Oct 15 '18 at 9:41
  • $\begingroup$ Sry, I was too unspecific. I refer to the reconstruction loss of the VAE. In this Keras example the user also has a choice between BCE and MSE. $\endgroup$ – Tik0 Oct 15 '18 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.