12
$\begingroup$

There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?

$\endgroup$
16
$\begingroup$

Here's a worked example in the case of iid binary data, each with a success/failure recorded as $y_i \in \{0,1\}$.

For labels $y_i\in \{0,1\}$, the likelihood of some binary data under the Bernoulli model with parameters $\theta$ is $$ \mathcal{L}(\theta) = \prod_{i=1}^n p(y_i=1|\theta)^{y_i}p(y_i=0|\theta)^{1-y_i}\\ $$ whereas the log-likelihood is $$ \log\mathcal{L}(\theta) = \sum_{i=1}^n y_i\log p(y=1|\theta) + (1-y_i)\log p(y=0|\theta) $$

And the binary cross-entropy is $$ L(\theta) = -\frac{1}{n}\sum_{i=1}^n y_i\log p(y=1|\theta) + (1-y_i)\log p(y=0|\theta) $$

Clearly, $ \log \mathcal{L}(\theta) = -nL(\theta) $.

We know that an optimal parameter vector $\theta^*$ is the same for both because we can observe that for any $\theta$ which is not optimal, we have $\frac{1}{n} L(\theta) > \frac{1}{n} L(\theta^*)$, which holds for any $\frac{1}{n} > 0$. (Remember, we want to minimize cross-entropy, so the optimal $\theta^*$ has the least $L(\theta^*)$.)

Likewise, we know that the optimal value $\theta^*$ is the same for $\log \mathcal{L}(\theta)$ and $ \mathcal{L}(\theta)$ because $\log(x)$ is a monotonic increasing function for $x \in \mathbb{R}^+$, so we can write $\log \mathcal{L}(\theta) < \log\mathcal{L}(\theta^*)$. (Remember, we want to maximize the likelihood, so the optimal $\theta^*$ has the most $\mathcal{L}(\theta^*)$.)

Some sources omit the $\frac{1}{n}$ from the cross-entropy. Clearly, this only changes the value of $L(\theta)$, but not the location of the optima, so from an optimization perspective the distinction is not important. The negative sign, however, is obviously important since it is the difference between maximizing and minimizing!


Some more additional examples and more general result can be found in this related thread: How to construct a cross-entropy loss for general regression targets?

$\endgroup$
7
  • $\begingroup$ Is this specific for logistic regression or applicable in general such as to a deep/convolutional/recurrent neural network? $\endgroup$ – Dave Jul 14 '20 at 14:55
  • $\begingroup$ The definition of the loss/MLE function doesn't change -- as you can see, the likelihood is not tied to any particular functional form of the model -- so we can infer that cross-entropy loss and the binomial MLE are the same in both logistic regression and NNs. From an optimization perspective, the point of departure is that logistic regression is strongly convex under certain assumptions, while neural networks are generally non-convex. For example, you can permute the weights of a neural network and obtain the same loss, so there are many $\theta^*$ values with the same loss. $\endgroup$ – Sycorax Jul 14 '20 at 14:58
  • $\begingroup$ Same loss or same minimum loss? $\endgroup$ – Dave Jul 14 '20 at 15:02
  • $\begingroup$ Permutation equivalence of weights in an NN implies that if you have obtained some loss $L(\theta)$, then permuting the weights in a particular way will give the same loss value for the permuted vector $\theta$. So under these kinds of permutations, the losses are equal. This is true for minima corresponding to $\theta^*$ as well as the general case $\theta$ which do not minimize $L(\theta)$. $\endgroup$ – Sycorax Jul 14 '20 at 15:05
  • 1
    $\begingroup$ @gebbissimo It depends on what you mean. I'm not aware of a probability model which results in this loss function. On the other hand, this post explains why using this loss with non-binary targets in [0,1] are still useful models for training a neural network, without appealing to a probability model. stats.stackexchange.com/questions/394582/… $\endgroup$ – Sycorax Oct 2 '20 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.