Difference between Empirical distribution and Bernoulli distribution

Question

I've been studying binary cross entropy error for binary classification weight optimization.

From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with the same set of outcomes in terms of average bits. It is simply a Kullback-Leibler divergence (or relative entropy) but without additive term of entropy of first probability distribution. Thus if we had two probability distributions $p$ and $q$, the cross entropy would be:

$$H(p,q) = H(p) + D_{KL}(p||q)$$

and therefore relative entropy would be:

$$D_{KL}(p||q)=H(p,q)-H(p)$$

---

Confusion arises when these $p$ and $q$ have to be chosen. Usually for binary cross entropy, $p$ and $q$ must be Bernoulli distributions (where probability is $p$ for the sample $1$ and $p-1$ for the sample $0$). Then the cross entropy is measured between these distributions. Considering that probability mass function of Bernoulli distribution only takes $0$ and $1$ as a domain, it seems to be optimal choice for binary classification.

But there is an equivalent to this, where the cross entropy is taken for empirical distribution of the actual value and predicted distribution. In this case, empirical distribution must take $y$ and $y-1$ (where $y$ is the actual value) and compare it to the predicted $\hat{y}$ and $\hat{y}-1$.

From my understanding, empirical distribution function is basically assigning the probability of 1 if the value in sample belongs to specific class and 0 if not.

---

Why is cross entropy between Bernoulli distribution of the actual value and Bernoulli distribution of predicted value equivalent to the cross entropy empirical distribution of actual value and predicted distribution?

More generally, how are empirical distribution function and Bernoulli distribution similar in the specific case of binary cross entropy?

Thank you!

Answer 1

More generally, how are empirical distribution function and Bernoulli distribution similar in the specific case of binary cross entropy?

Bernoulli distribution is an abstract, parametric function, empirical distribution is a nonparametric estimator of the underlying distribution function (no matter what it is).

If your data consists of only two possible values, then it can be thought as a sample from Bernoulli distribution $p$, so that probability of observing $1$ is $p$ and probability of observing $0$ is $1-p$. The maximum likelihood estimator of $p$ is the proportion of $1$ in your sample. At the same time, empirical distribution of such sample for each value will assign probability equal to the proportion of of such value in your sample, so it will be exactly the same as Bernoulli distribution with $p$ estimated using maximum likelihood.