Why binary crossentropy can be used as the loss function in autoencoders? [duplicate]

Question

I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray classification problems where the ground-truth labels (i.e. $y$) are either 0 or 1 and therefore when predictions (i.e. $p$) are correct, in both cases, the loss value would be zero:

$$ BCE(y,p) = -y\cdot\log(p) - (1-y)\cdot\log{(1-p)} $$

$$ BCE(0,0) = 0, \hspace{2mm} BCE(1,1) = 0 $$

However, binary crossentropy does not have a value of zero when neither of its arguments are both zero or one, which is the case for an autoencoder with ground-truth labels in range $[0,1]$ (i.e. assuming the input data has been normalized in this range). I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. What am I missing here?

Answer 1

I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same.

That's exactly the misconception you have. You think that in order for a loss function to be used in a model like an autoencoder, it must have a value of zero when predictions equal to true labels. That's simply wrong since in most of the machine learning models (including autoencoders) we are trying to minimize a loss/cost function. And we are doing this with the assumption that the loss function we are using when reaches its minimum point, implies that the predictions and true labels are the same. That's the condition for using a function as a loss function in a model trained based on minimzing loss function. Note that the value of loss function at this minimum point may not be zero at all, however we don't care about this as long as it implies in that point predictions and true labels are the same.

Now let's verify this is the case for binary crossentropy: we need to show that when we reach the minimum point of binary crossentropy it implies that $y = p$, i.e. predictions equal to true labels. To find the minimum point, we take the derivative with respect to $p$ and set it equal to zero (note that in the following calculations I have assumed that the $\log$ is natural logarithm function to make calculations a little easier):

$$\begin{align}&\frac{\partial BCE(y,p)}{\partial p} = 0\\ &\implies -y\cdot\dfrac{1}{p} - (1-y)\cdot\dfrac{-1}{1-p} = 0\\ &\implies -y\cdot(1-p) + (1-y)\cdot p = 0\\ &\implies -y + y\cdot p + p - y\cdot p = 0\\ &\implies p - y = 0\\ &\implies y = p \end{align}$$

Answer 2

As @today pointed out, loss value doesn't have to be 0 when the solution is optimal, it is enough that it is minimal.

One thing I would like to add is why one would prefer binary crossentropy over MSE. Normally, the activation function of the last layer is sigmoid, which can lead to loss saturation ("plateau"). This saturation could prevent gradient-based learning algorithms from making progress. In order to avoid it, it is then good to have a log in the objective function to undo the exp in sigmoid, and this is why binary crossentropy is preferred (because it uses log, unlike MSE). I have read this in the Deep Learning Book, but I now can't find where exactly (I think chapter 8).