Relationship between cross entropy and average negative log likelihood

Question

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood".

To start, I already fully understand these definitions:

1. Entropy of a probability distribution $p$ with $K$ classes:

$$ H(p) = - \sum_{k=1}^{K} p_k \log p_k $$

2. Cross entropy between two probability distributions $p$ (ground-truth) and $q$ (predicted):

$$ H(p, q) = - \sum_{k=1}^{K} p_k \log q_k $$

My specific confusion comes from reading Kevin Murphy's 2021 book "Probabilistic Machine Learning: An Introduction". He says something like this about Kullback-Leibler divergence (it's a paraphrase summarization of sections 4.2 and 6.2):

$KL(p||q) = \sum_{k=1}^{K} p_k \log p_k - \sum_{k=1}^{K} p_k \log q_k$

We recognize the first term as the negative entropy and the second term as the cross entropy. The first term is a constant with respect to our predictions $q$, so we can ignore it.

Let us suppose the $p$ distribution is defined with a delta function $\delta$ like this: $ p(x) = \frac{1}{N} \sum_{n=1}^{N} \delta(x - x_n)$ .

Then the KL divergence becomes \begin{align} KL(p||q) &= -H(p) - \frac{1}{N} \sum_{n=1}^{N} \log q(y_n)\\ &= constant + NLL \end{align} This is called the cross entropy objective, and is equal to the average negative log likelihood of q on the training set.

Questions:

1. The term $\frac{1}{N} \sum_{n=1}^{N} \log q(y_n)$ mentions one distribution $q$. How can it be a cross-entropy term when cross entropy is defined for two distributions $p$ and $q$?

2. How does a log-likehood expression in terms of $N$ training instances ($\frac{1}{N} \sum_{n=1}^{N})$ turn into a cross-entropy expression in terms of $K$ classes ($\sum_{k=1}^{K}$)?

3. Is the author's use of a delta function $\delta$ just another way of saying a one-hot distribution?

I'm still confused even after reading other posts like this one, this one, and this one.

Answer 1

1. This is because of the claim about delta distributions. Now $p_k$ is $0$ for all $k$ but one. Those $p_k \log q_k$ terms in the sum over $k$ can be skipped. For the other, $p_k$ is $1$, so there’s no need to write that you multiply by $1$.
2. I think this is answered in (1). The $0$-valued terms are skipped. In a sense, you have a double sum over $n$ and over $k$, but then the sum over $k$ goes away.
3. Yes, that’s right. “One-hot” is terminology from the area of digital circuits; the delta function is from mathematics.