10
$\begingroup$

I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray classification problems where the ground-truth labels (i.e. $y$) are either 0 or 1 and therefore when predictions (i.e. $p$) are correct, in both cases, the loss value would be zero:

$$ BCE(y,p) = -y.log(p) - (1-y).\log{(1-p)} $$

$$ BCE(0,0) = 0, BCE(1,1) = 0 $$

However, binary crossentropy does not have a value of zero when neither of its arguments are both zero or one, which is the case for an autoencoder with ground-truth labels in range $[0,1]$ (i.e. assuming the input data has been normalized in this range). I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. What am I missing here?

$\endgroup$
7
  • 1
    $\begingroup$ NOTE FOR CLOSE VOTERS (i.e. claiming this to be duplicate of this question): 1) It's a very weird decision to close an older question (i.e. this) as a duplicate of a newer question, and 2) Although these two questions have the same title, they attempt to ask different questions: this one asks why BCE works for autoencoders in the first place (and its answer provide a proof), the other one asks why it might be better or worse to use this loss function in autoencoders. $\endgroup$
    – today
    Jul 30 '20 at 22:29
  • $\begingroup$ (1) There's no particular reason to requires duplicates be ordered temporally, and this has never been a requirement here. (2) If we exclude the title, which you concede is the same, then this post has only 2 questions. (a) "Why BCE can be used as a loss function on images?" which repeats the title and (b) "What am I missing here?" which, in context, doesn't read as distinct from (a). The answer shows that BCE attains 0 loss when $y=p$, but this isn't a distinguishing feature of BCE loss from any other loss. On the other hand, the other post shows how BCE loss behaves. $\endgroup$
    – Sycorax
    Jul 30 '20 at 22:58
  • $\begingroup$ If the title & body were edited, perhaps to ask something like "When targets are non-binary, at what value is BCE loss minimized?" then it would not be a duplicate of the other post. Given that you've accepted an answer which is directly responsive to this question, this rephrasing seems to capture what you want to know. $\endgroup$
    – Sycorax
    Jul 30 '20 at 23:01
  • $\begingroup$ Anyway, the place for discussing this is meta.stats.SE $\endgroup$
    – Sycorax
    Jul 30 '20 at 23:12
  • 1
    $\begingroup$ 2/ Pardon me, but I think you haven't read the other question carefully at all because the most funny thing is that the OP of the other question has linked to one of my answers on SO for the same proof which I have provided in that answer. So clearly that question is build upon this question. Anyways, I no longer want to continue this discussion because it's futile and does not worth it. Thanks for your replies, though (it's very good to see that you are not one of those silent downvote/close voters on SE network). $\endgroup$
    – today
    Aug 1 '20 at 11:20
6
$\begingroup$

I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same.

That's exactly the misconception you have. You think that in order for a loss function to be used in a model like an autoencoder, it must have a value of zero when predictions equal to true labels. That's simply wrong since in most of the machine learning models (including autoencoders) we are trying to minimize a loss/cost function. And we are doing this with the assumption that the loss function we are using when reaches its minimum point, implies that the predictions and true labels are the same. That's the condition for using a function as a loss function in a model trained based on minimzing loss function. Note that the value of loss function at this minimum point may not be zero at all, however we don't care about this as long as it implies in that point predictions and true labels are the same.

Now let's verify this is the case for binary crossentropy: we need to show that when we reach the minimum point of binary crossentropy it implies that $y = p$, i.e. predictions equal to true labels. To find the minimum point, we take the derivative with respect to $p$ and set it equal to zero (note that in the following calculations I have assumed that the $log$ is natural logarithm function to make calculations a little easier):

$$\begin{align}&\frac{\partial BCE(y,p)}{\partial p} = 0\\ &\implies -y.\dfrac{1}{p} - (1-y).\dfrac{-1}{1-p} = 0\\ &\implies -y.(1-p) + (1-y).p = 0\\ &\implies -y + y.p + p - y.p = 0\\ &\implies p - y = 0\\ &\implies y = p \end{align}$$

$\endgroup$
8
  • 2
    $\begingroup$ Could you also point out why one should use binary crossentropy (BC) over e.g. mean-squared-error (MSE) at all? In fact they both have the same minimum so they should be equivalent. However, the outputs (e.g. VAE on MNIST) look different. $\endgroup$
    – Tik0
    Oct 15 '18 at 8:47
  • $\begingroup$ @Tik0 I don't think VAE is trained using either of MSE or BCE loss functions. Instead, KL-divergence is usually used as the loss function in this specific type of autoencoders. If you have any example of autoencoder trained using MSE and BCE loss and there is a noticable difference between the results obtained, please provide a reference so that I can take a look at it and investigate more. $\endgroup$
    – today
    Oct 15 '18 at 9:41
  • $\begingroup$ Sry, I was too unspecific. I refer to the reconstruction loss of the VAE. In this Keras example the user also has a choice between BCE and MSE. $\endgroup$
    – Tik0
    Oct 15 '18 at 10:29
  • 1
    $\begingroup$ You've shown that BCE is minimized when $y=p$. But there are many losses which are minimized in this way: $(y-p)^2$, $| y - p |$, $\log \cosh (y-p)$, etc. Given that all of these models have the same minima, which criteria would guide you to choose one loss over another? In other words, why is BCE is a good choice for the auto-encoder task which is named as a motivation in OP? $\endgroup$
    – Sycorax
    Jul 28 '20 at 3:10
  • 1
    $\begingroup$ @Sycorax I respect your decision to close this question as a duplicate of the other question, especially for the fact that you are an experienced user. But these two questions are not the same, but only in their titles. It would have been much wiser to edit at least the tile of the other question to clarify this. Also read my recent comment under this question. Anyways, as I mentioned earlier, I respect your decision (though, I stand by my points which I tried to clarify on multiple occasions regarding this question and its answer). $\endgroup$
    – today
    Jul 30 '20 at 22:47
3
$\begingroup$

As @today pointed out, loss value doesn't have to be 0 when the solution is optimal, it is enough that it is minimal.

One thing I would like to add is why one would prefer binary crossentropy over MSE. Normally, the activation function of the last layer is sigmoid, which can lead to loss saturation ("plateau"). This saturation could prevent gradient-based learning algorithms from making progress. In order to avoid it, it is then good to have a log in the objective function to undo the exp in sigmoid, and this is why binary crossentropy is preferred (because it uses log, unlike MSE). I have read this in the Deep Learning Book, but I now can't find where exactly (I think chapter 8).

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.