I saw some examples of Autoencoders (on images) which use sigmoid as output layer and BinaryCrossentropy as loss function.

The input to the Autoencoder is normalized $[0..1]$. The sigmoid outputs values (value of each pixel of the image) $[0..1]$

I tried to evaluate the output of BinaryCrossentropy and I'm confused.

Assume for simplicity we have a [2x2] image and we run Autoencoder and get 2 results. One result is close to the True value and the second is same as the true value:

import numpy as np
import tensorflow as tf
bce = tf.keras.losses.BinaryCrossentropy()
y_true = [0.5, 0.3, 0.5, 0.9]
y_pred = [0.1, 0.3, 0.5, 0.8]
print(bce(y_true, y_pred).numpy())

y_pred = [0.5, 0.3, 0.5, 0.9]
print(bce(y_true, y_pred).numpy())



As you can see, the second example (which is the same as the true value) gets a low score (which is ok, low score of loss function, but it is not zero and its not far away from the first example).

It seems that using BinaryCrossentropy as loss function won't give us the best results.

Will the best value be close to 0.5?

What am I missing?

  • $\begingroup$ As a side note, be careful when using binary cross-entropy in Keras. Depending on which metrics you are using Keras may infer that your metric is binary i.e. only observe the first element of the output. This won't affect your loss, but will affect your metrics, so when using model.fit make sure you pass an actual instance of the desired metric(s) to model.compile instead of referring to the metric by a string of the metric name. $\endgroup$
    – Avelina
    Jun 28 at 15:39

The main idea is that loss functions don't need to attain a 0 at their minimum. To see why, consider some loss function that we know attains a minimum at 0 such as $L(\hat{y})=(y-\hat{y})^2$. Clearly, this attains 0 when $y = \hat{y}$. But this model is precisely the same as $F(\hat{y})=(y-\hat{y})^2 + c$ for some fixed $c>0$. Even though $F$ never attains 0, it should be obvious that the it still attains its minimum at $y = \hat{y}$, the same as for $L$.

We can verify that $F$ and $L$ have the same minima by taking the derivatives and showing that when $F'=0$, we have $F'' >0$, likewise for $L$, and then comparing them to see that they are the same.

In the binary cross-entropy case, we can repeat the same procedure. But first, let's just inspect the function to see what we can learn. We have the function

$$g(\hat{y})=-y \log(\hat{y}) - (1-y)\log(1-\hat{y})$$

  • For this problem, $y \in (0,1)$, $y$ is fixed, and $\hat{y} \in (0,1)$.

  • We know that $-\log(x)$ is strictly decreasing for real $x>0$ (or we can check the first derivative to verify).

  • We know that $-\log(1)=0$.

Putting it all together, we know that for every $x \in (0,1)$, we know that $-\log(x)$ must be positive, because it's a strictly decreasing function and only attains 0 at 1. (Or you can draw a graph of $-\log(x)$). Referring back to $g$, we know that the loss must also be strictly positive because we're considering solely the multiplication of positive numbers and the addition of positive numbers, which must yield positive numbers. No matter the specific values of $y,\hat{y}$, we've already shown that all loss values of $g$ must be positive!

Finally, we can show that given $y$, the prediction $\hat{y}$ which minimizes $g$ is $y = \hat{y}$.

  • First order conditions: $$\begin{align} g'(\hat{y}) =0 &= \frac{y-\hat{y}}{(1-\hat{y})\hat{y}} \\ \hat{y}&=y \end{align} $$

  • Second order conditions:

$$\begin{align} g''(\hat{y}) &= \frac{\hat{y}^2-2y\hat{y}+\hat{y}}{\hat{y}^2(\hat{y}-1)^2 } \\ &= \frac{ y^2 - 2y^2+y }{y^2(y-1)^2 } \\ &= \frac{ -y(y-1) }{y^2(y-1)^2 } \\ &= \frac{ -y }{y^2(y-1) } \\ \end{align}$$

We are examining the point $y=\hat{y}$, which lets us simplify the expression. Also, we know $y < 1$, so the denominator must be negative, and we know $y >0$ so the numerator must also be negative, therefore $g''(\hat{y})>0$ at $y=\hat{y}$, so this point is a minimum of $g$.

Does the best value will be close to 0.5 ?

Because we know $y = \hat{y}$ is the minimum, we know that $\hat{y}=0.5$ is only an optimal prediction when $y=0.5$.

We can use your code to demonstrate the same.

import numpy as np
import tensorflow as tf

bce = tf.keras.losses.BinaryCrossentropy()

y_true = [0.5, 0.3, 0.5, 0.9]
y_pred1 = [0.5, 0.5, 0.5, 0.5]
loss1 = bce(y_true, y_pred1).numpy()

y_pred2 = y_true
loss2 = bce(y_true, y_pred2).numpy()
loss1 > loss2 # True 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.