I've been studying binary cross entropy error for binary classification weight optimization.

From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with the same set of outcomes in terms of average bits. It is simply a Kullback-Leibler divergence (or relative entropy) but without additive term of entropy of first probability distribution. Thus if we had two probability distributions $p$ and $q$, the cross entropy would be:

$$H(p,q) = H(p) + D_{KL}(p||q)$$

and therefore relative entropy would be:


Confusion arises when these $p$ and $q$ have to be chosen. Usually for binary cross entropy, $p$ and $q$ must be Bernoulli distributions (where probability is $p$ for the sample $1$ and $p-1$ for the sample $0$). Then the cross entropy is measured between these distributions. Considering that probability mass function of Bernoulli distribution only takes $0$ and $1$ as a domain, it seems to be optimal choice for binary classification.

But there is an equivalent to this, where the cross entropy is taken for empirical distribution of the actual value and predicted distribution. In this case, empirical distribution must take $y$ and $y-1$ (where $y$ is the actual value) and compare it to the predicted $\hat{y}$ and $\hat{y}-1$.

From my understanding, empirical distribution function is basically assigning the probability of 1 if the value in sample belongs to specific class and 0 if not.

Why is cross entropy between Bernoulli distribution of the actual value and Bernoulli distribution of predicted value equivalent to the cross entropy empirical distribution of actual value and predicted distribution?

More generally, how are empirical distribution function and Bernoulli distribution similar in the specific case of binary cross entropy?

Thank you!


1 Answer 1


More generally, how are empirical distribution function and Bernoulli distribution similar in the specific case of binary cross entropy?

Bernoulli distribution is an abstract, parametric function, empirical distribution is a nonparametric estimator of the underlying distribution function (no matter what it is).

If your data consists of only two possible values, then it can be thought as a sample from Bernoulli distribution $p$, so that probability of observing $1$ is $p$ and probability of observing $0$ is $1-p$. The maximum likelihood estimator of $p$ is the proportion of $1$ in your sample. At the same time, empirical distribution of such sample for each value will assign probability equal to the proportion of of such value in your sample, so it will be exactly the same as Bernoulli distribution with $p$ estimated using maximum likelihood.

  • $\begingroup$ Thank you for the answer! You mentioned maximum likelihood estimator of $p$, but between what values (other than $p$) is likelihood maximized? Apologies for my confusion. $\endgroup$
    – ShellRox
    Jul 18, 2018 at 15:20
  • 1
    $\begingroup$ @ShellRox I'm afraid, I don't understand your question. Maximum likelihood = such parameter $p$ that maximizes the likelihood function over your data. $\endgroup$
    – Tim
    Jul 18, 2018 at 15:26
  • $\begingroup$ So from my understanding, It's empirical distribution function being equal to the proportion of observation from the Bernoulli distribution sample being less than or equal to the point. Thank you! $\endgroup$
    – ShellRox
    Jul 18, 2018 at 15:54
  • $\begingroup$ I think you're playing it a bit loose with the definition of a Bernoulli distribution. A Bernoulli r.v. has 3 defining attributes (1) dichotomous outcomes (2) iid experiments and (3) a fixed probability of "success". Your explanation seems to suggest that only (1) is required for a random variable to be a Bernoulli r.v. $\endgroup$
    – Sycorax
    Jul 18, 2018 at 16:43
  • $\begingroup$ @Sycorax yes, but I really don't think that the two other features make any difference in terms of the question that was asked. $\endgroup$
    – Tim
    Jul 18, 2018 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.