A recent paper argues [1] that, no, cross-entropy is not adequate for $y \in [0,1]$, because it is not a Bernoulli distributed variable. While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the Bernoulli likelihood.

A Beta distribution could be used instead, but they also propose a new distribution that augments the Bernoulli, which entails a simple correction to cross-entropy.

The Continuous Bernoulli distribution is given by, with $\lambda \in (0,1)$, $x \in [0,1]$:

$$p_{\mathcal{CB}}(x|\lambda) = C(\lambda)\lambda^x(1-\lambda)^{1-x}$$

Contrast it with the original Bernoulli, with $p \in (0,1)$, $ k \in \{0,1\} $:

$$p_{\mathcal{B}}(k|p) = p^k(1-p)^{1-k}$$

The Continuous Bernoulli is proportional to the Bernoulli, but with continuous $k$, and the correction term is introduced to make it a valid distribution.

The new cross-entropy then is:

$$\mathcal L(\hat y, y) = y\log(\hat y) + (1 - y) \log(1-\hat y) + \color{red}{\log C(\hat y)}$$

This last term, the normalizing correction, is given by:

$$C(x) = \begin{cases} \begin{align} &\frac{2\tanh^{-1}(1-2x)}{1-2x} \quad &\text{if} \quad x \neq 0.5\\ &2 \quad &\text{if} \quad x = 0.5 \end{align} \end{cases}$$

_{[1] Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).}

If we consider a continuous relaxation of Bernoulli that allows the true probability to be between 0 and 1, a recent paper argues [1] that, no, cross-entropy is not adequate for $y \in [0,1]$, because it is not a Bernoulli distributed variable. While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the Bernoulli likelihood. The continuous $y$ can be regarded as a soft-label.

A Beta distribution could be used instead, but they also propose a new distribution that augments the Bernoulli, which entails a simple correction to cross-entropy.

The Continuous Bernoulli distribution is given by, with $\lambda \in (0,1)$, $x \in [0,1]$:

$$p_{\mathcal{CB}}(x|\lambda) = C(\lambda)\lambda^x(1-\lambda)^{1-x}$$

Contrast it with the original Bernoulli, with $p \in (0,1)$, $ k \in \{0,1\} $:

$$p_{\mathcal{B}}(k|p) = p^k(1-p)^{1-k}$$

The Continuous Bernoulli is proportional to the Bernoulli, but with continuous $k$, and the correction term is introduced to make it a valid distribution.

The new cross-entropy then is:

$$\mathcal L(\hat y, y) = y\log(\hat y) + (1 - y) \log(1-\hat y) + \color{red}{\log C(\hat y)}$$

This last term, the normalizing correction, is given by:

$$C(x) = \begin{cases} \begin{align} &\frac{2\tanh^{-1}(1-2x)}{1-2x} \quad &\text{if} \quad x \neq 0.5\\ &2 \quad &\text{if} \quad x = 0.5 \end{align} \end{cases}$$

_{[1] Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).}

A recent paper argues [1] that, no, cross-entropy is not adequate for $y \in [0,1]$, because it is not a Bernoulli distributed variable. While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the Bernoulli likelihood.

A Beta distribution could be used instead, but they also propose a new distribution that augments the Bernoulli, which entails a simple correction to cross-entropy.

The Continuous Bernoulli distribution is given by, with $\lambda \in (0,1)$:

$$p_{\mathcal{CB}}(x|\lambda) = C(\lambda)\lambda^x(1-\lambda)^{1-x}$$

Contrast it with the original Bernoulli, with $ k \in \{0,1\} $:

$$p_{\mathcal{B}}(x|\lambda) = p^k(1-p)^{1-k}$$

The Continuous Bernoulli is proportional to the Bernoulli, but with continuous $k$, and the correction term is introduced to make it a valid distribution.

The new cross-entropy then is:

$$\mathcal L(\hat y, y) = y\log(\hat y) + (1 - y) \log(1-\hat y) + \color{red}{\log C(\hat y)}$$

This last term, the normalizing correction, is given by:

$$C(x) = \begin{cases} \begin{align} &\frac{2\tanh^{-1}(1-2x)}{1-2x} \quad &\text{if} \quad x \neq 0.5\\ &2 \quad &\text{if} \quad x = 0.5 \end{align} \end{cases}$$

_{[1] Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).}

A recent paper argues [1] that, no, cross-entropy is not adequate for $y \in [0,1]$, because it is not a Bernoulli distributed variable. While their work is concerned with Variational Autoencoders, the argument can be extended to other uses of the Bernoulli likelihood.

A Beta distribution could be used instead, but they also propose a new distribution that augments the Bernoulli, which entails a simple correction to cross-entropy.

The Continuous Bernoulli distribution is given by, with $\lambda \in (0,1)$, $x \in [0,1]$:

$$p_{\mathcal{CB}}(x|\lambda) = C(\lambda)\lambda^x(1-\lambda)^{1-x}$$

Contrast it with the original Bernoulli, with $p \in (0,1)$, $ k \in \{0,1\} $:

$$p_{\mathcal{B}}(k|p) = p^k(1-p)^{1-k}$$

The Continuous Bernoulli is proportional to the Bernoulli, but with continuous $k$, and the correction term is introduced to make it a valid distribution.

The new cross-entropy then is:

$$\mathcal L(\hat y, y) = y\log(\hat y) + (1 - y) \log(1-\hat y) + \color{red}{\log C(\hat y)}$$

This last term, the normalizing correction, is given by:

$$C(x) = \begin{cases} \begin{align} &\frac{2\tanh^{-1}(1-2x)}{1-2x} \quad &\text{if} \quad x \neq 0.5\\ &2 \quad &\text{if} \quad x = 0.5 \end{align} \end{cases}$$

_{[1] Loaiza-Ganem, G., & Cunningham, J. P. (2019). The continuous Bernoulli: fixing a pervasive error in variational autoencoders. In Advances in Neural Information Processing Systems (pp. 13266-13276).}