# Can we derive cross entropy formula as maximum likelihood estimation for SOFT LABELS?

For hard integer labels {0,1}, the cross entropy simplifies to the log loss. In this case, it is easy to show that minimizing the cross entropy is equivalent to maximizing the log likelihood, see e.g. https://stats.stackexchange.com/a/364237/179312

Can we also show this for soft float labels [0,1]? This thread states that the cross entropy function is also appropriate here. But how does the log likelihood function look like in this case?

Soft labels define a 'true' target distribution over class labels for each data point. As I described previously, a probabilistic classifier can be fit by minimizing the cross entropy between the target distribution and the predicted distribution. In this context, minimizing the cross entropy is equivalent to minimizing the KL divergence. So, what we're doing is finding a good approximation to the target distribution (as measured by KL divergence). However, as described below, the problem can equivalently be cast as a weighted maximum likelihood problem, where the soft labels determine the weights. I'll show this for binary classification, but the same reasoning also applies to multiclass problems.

### Probabilistic binary classification with soft labels

Let $$X = \{x_1, \dots, x_n\}$$ be a set of data points with binary class labels $$\mathbf{y} \in \{0, 1\}^n$$. Assume the class labels are conditionally independent, given $$X$$. The class labels are unknown, but we have soft labels $$\mathbf{\ell} \in [0,1]^n$$, where $$\ell_i$$ gives the probability that $$y_i=1$$. The soft labels define a Bernoulli target distribution over class labels for each data point:

$$p(y \mid \ell_i) = \left\{ \begin{array}{cl} \ell_i & y = 1 \\ 1 - \ell_i & y = 0 \\ \end{array} \right.$$

The goal is to learn a conditional distribution $$q(y \mid x, \theta)$$ (a.k.a. probabilistic classifier, parameterized by $$\theta$$), such that the predicted class probabilities approximate those given by the soft labels. We do this by minimizing the cross entropy between the target and predicted distributions over class labels, summed over data points:

$$\min_\theta \ \sum_{i=1}^n H \Big( p(y \mid \ell_i), q(y \mid x_i, \theta) \Big) \tag{1}$$

Writing out the expression for the cross entropy, the problem is:

$$\min_\theta \ -\sum_{i=1}^n \ell_i \log q(y=1 \mid x_i, \theta) - \sum_{i=1}^n (1-\ell_i) \log q(y=0 \mid x_i, \theta) \tag{3}$$

### Equivalence to weighted maximum likelihood

Suppose we define a new dataset $$(\tilde{X}, \tilde{\mathbf{y}})$$ by duplicating each data point. We assign hard class label $$1$$ to the first duplicate, and $$0$$ to the second duplicate. Furthermore, we assign a weight to each new data point. The first duplicates are weighted by the soft labels, and the second duplicates are weighted by one minus the soft labels. That is:

$$\begin{array}{ccl} \tilde{X} & = & \{x_1, \dots, x_n, x_1, \dots, x_n\} \\ \tilde{y} & = & [1, \dots, 1, 0, \dots, 0]^T \\ \tilde{w} & = & [\ell_1, \dots, \ell_n, 1-\ell_1, \dots, 1-\ell_n]^T \end{array} \tag{4}$$

Intuitively, you can think of the weights as a continuous analog of 'how many times' we've seen each case. We've constructed the new dataset in a way that translates soft labels into 'replications'. For example, if a point has soft label $$0.75$$, this is like seeing the same point three times with hard label $$1$$ and once with hard label $$0$$ (giving weights .75 and .25, respectively).

As above, we want to learn a conditional distribution $$q(y \mid x, \theta)$$, but this time using the new dataset with hard labels and weights. We do this by maximizing the weighted likelihood:

$$L_{\tilde{w}}(\theta; \tilde{X}, \tilde{\mathbf{y}}) = \prod_{i=1}^{2 n} q(\tilde{y}_i \mid \tilde{x}_i, \theta)^{\tilde{w}_i} \tag{5}$$

This is equivalent to minimizing the weighted negative log likelihood:

$$-\log L_{\tilde{w}}(\theta; \tilde{X}, \tilde{\mathbf{y}}) = -\sum_{i=1}^{2 n} \tilde{w}_i \log q(\tilde{y}_i \mid \tilde{x}_i, \theta) \tag{6}$$

Substitute in our expressions for $$\tilde{X}, \tilde{\mathbf{y}}, \tilde{w}$$:

$$\begin{matrix} -\log L_{\tilde{w}}(\theta; \tilde{X}, \tilde{\mathbf{y}}) = \\ -\sum_{i=1}^n \ell_i \log q(y=1 \mid x_i, \theta) - \sum_{i=1}^n (1-\ell_i) \log q(y=0 \mid x_i, \theta) \end{matrix}\tag{7}$$

The weighted negative log likelihood in $$(7)$$ is the same as the cross entropy loss in $$(3)$$. So, the weighted maximum likelihood problem here is equivalent to the cross entropy minimization problem above.

• Thank you very much for the detailed answer. I really like the idea of soft labels as replications. Still need to wrap my head around about how this "revisiting X times" translates to a fractional exponent in the weighted likelihood, but I hope to get there... – gebbissimo Oct 11 '20 at 16:30

If we consider a continuous relaxation of Bernoulli that allows the true probability to be between 0 and 1, a recent paper argues  that, no, cross-entropy is not adequate for $$y \in [0,1]$$, because it is not a Bernoulli distributed variable. While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the Bernoulli likelihood. The continuous $$y$$ can be regarded as a soft-label.

A Beta distribution could be used instead, but they also propose a new distribution that augments the Bernoulli, which entails a simple correction to cross-entropy.

The Continuous Bernoulli distribution is given by, with $$\lambda \in (0,1)$$, $$x \in [0,1]$$:

$$p_{\mathcal{CB}}(x|\lambda) = C(\lambda)\lambda^x(1-\lambda)^{1-x}$$

Contrast it with the original Bernoulli, with $$p \in (0,1)$$, $$k \in \{0,1\}$$:

$$p_{\mathcal{B}}(k|p) = p^k(1-p)^{1-k}$$

The Continuous Bernoulli is proportional to the Bernoulli, but with continuous $$k$$, and the correction term is introduced to make it a valid distribution.

The new cross-entropy then is:

$$\mathcal L(\hat y, y) = y\log(\hat y) + (1 - y) \log(1-\hat y) + \color{red}{\log C(\hat y)}$$

This last term, the normalizing correction, is given by:

C(x) = \begin{cases} \begin{align} &\frac{2\tanh^{-1}(1-2x)}{1-2x} \quad &\text{if} \quad x \neq 0.5\\ &2 \quad &\text{if} \quad x = 0.5 \end{align} \end{cases}

 Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).

• The paper points out that people sometimes erroneously treat values in $[0,1]$ as probabilities defining a Bernoulli distribution. I wholeheartedly agree with this--grayscale values are not probabilities of an image pixel being 0 or 1. They're continuous observed values and should be modeled as such. (continued...) – user20160 Oct 11 '20 at 4:39
• But, I'd argue this means that such values are not soft labels at all. I consider soft labels to be actual probabilities given for different values a discrete variable may take. These can arise, for example, when training one probabilistic model to approximate another. Minimizing the cross entropy is indeed an appropriate thing to do in this situation. – user20160 Oct 11 '20 at 4:39
• @user20160 Your logical is right, except that you conclusion "grayscale is not a probability, thus Continuous Bernoulli does not apply to soft labels" is frail. If you treat $k$ as continuous, as they did, you need to include the correction factor, there is no escape from that. – Firebug Oct 13 '20 at 12:06