Behaviour of Keras BinaryCrossentropy() loss function

Question

From the Keras source code, this is the definition of the BinaryCrossentropy() for the Numpy backend and the plot of the loss function for the values around logit 0 in both directions (appoaching to it from the sides):

import numpy as np

def sigmoid(x):
    return 1. / (1. + np.exp(-x))

def binary_crossentropy(target, output, from_logits=False):
    if not from_logits:
        output = np.clip(output, 1e-7, 1 - 1e-7)
        output = np.log(output / (1 - output))

    return (target * -np.log(sigmoid(output)) +
            (1 - target) * -np.log(1 - sigmoid(output)))

loss = binary_crossentropy(np.array([-1,-.5,-.1,0,.1,.5,1]), np.array([1,.5,.1,0,-.1,-.5,-1]), from_logits=True)
plt.plot(loss)

Could anyone help with understanding why the minimum of the loss function is at point 4 (0.1, -0.1) instead of 3 (0.0, 0.0) when both prediction and target values are the same?

Answer 1

Cross-entropy between two probability distributions $p$ and $q$ is defined as

$$ H(p, q) = E_p [-\log q] = -\sum_x p(x) \log q(x) $$

In machine learning, $q$'s are the predicted probabilities, while $p$'s are the observed labels in $\{0, 1\}$, the "true probabilities". Both $p$ and $q$ need to be in $(0, 1)$, so your targets are wrong, leading to the results you have show.

For different reasons (usually computational ones), people sometimes prefer their models to return log-odds (logits), i.e. values in $(-\infty, \infty)$, rather then transforming them to probabilities. This is why functions as the one above accepts logits as inputs. However this only applies to the predicted values, not the target values. If your target values were unconstrained real numbers, then this would be a regression problem and you would be using the loss functions appropriate for regression (squared error, absolute error, Huber loss, etc.). Cross-entropy would not have any mathematical sense if the values of $p$ were unconstrained real numbers, while it has information-theoretic meaning if the values are probabilities.