# Relationship between cross entropy and average negative log likelihood

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood".

To start, I already fully understand these definitions:

1. Entropy of a probability distribution $$p$$ with $$K$$ classes:

$$H(p) = - \sum_{k=1}^{K} p_k \log p_k$$

1. Cross entropy between two probability distributions $$p$$ (ground-truth) and $$q$$ (predicted):

$$H(p, q) = - \sum_{k=1}^{K} p_k \log q_k$$

My specific confusion comes from reading Kevin Murphy's 2021 book "Probabilistic Machine Learning: An Introduction". He says something like this about Kullback-Leibler divergence (it's a paraphrase summarization of sections 4.2 and 6.2):

$$KL(p||q) = \sum_{k=1}^{K} p_k \log p_k - \sum_{k=1}^{K} p_k \log q_k$$

We recognize the first term as the negative entropy and the second term as the cross entropy. The first term is a constant with respect to our predictions $$q$$, so we can ignore it.

Let us suppose the $$p$$ distribution is defined with a delta function $$\delta$$ like this: $$p(x) = \frac{1}{N} \sum_{n=1}^{N} \delta(x - x_n)$$ .

Then the KL divergence becomes \begin{align} KL(p||q) &= -H(p) - \frac{1}{N} \sum_{n=1}^{N} \log q(y_n)\\ &= constant + NLL \end{align} This is called the cross entropy objective, and is equal to the average negative log likelihood of q on the training set.

Questions:

1. The term $$\frac{1}{N} \sum_{n=1}^{N} \log q(y_n)$$ mentions one distribution $$q$$. How can it be a cross-entropy term when cross entropy is defined for two distributions $$p$$ and $$q$$?

2. How does a log-likehood expression in terms of $$N$$ training instances ($$\frac{1}{N} \sum_{n=1}^{N})$$ turn into a cross-entropy expression in terms of $$K$$ classes ($$\sum_{k=1}^{K}$$)?

3. Is the author's use of a delta function $$\delta$$ just another way of saying a one-hot distribution?

I'm still confused even after reading other posts like this one, this one, and this one.

1. This is because of the claim about delta distributions. Now $$p_k$$ is $$0$$ for all $$k$$ but one. Those $$p_k \log q_k$$ terms in the sum over $$k$$ can be skipped. For the other, $$p_k$$ is $$1$$, so there’s no need to write that you multiply by $$1$$.
2. I think this is answered in (1). The $$0$$-valued terms are skipped. In a sense, you have a double sum over $$n$$ and over $$k$$, but then the sum over $$k$$ goes away.
• For 1, I know that to compute total cross-entropy loss over a data set, you compute the mean cross-entropy loss over each training instances, like $H_{total}(P, Q) = -\frac{1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} p_{k,n} \log q_{k,n}$ for class $k$ and training instance $n$. So the relation between cross-entropy and negative log likelihood is that the true distribution $p$ is implicitly one-hot, so you can remove the summation over $K$ ? Is that all there is??? – stackoverflowuser2010 May 20 at 20:03