I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood".

To start, I already fully understand these definitions:

  1. Entropy of a probability distribution $p$ with $K$ classes:

$$ H(p) = - \sum_{k=1}^{K} p_k \log p_k $$

  1. Cross entropy between two probability distributions $p$ (ground-truth) and $q$ (predicted):

$$ H(p, q) = - \sum_{k=1}^{K} p_k \log q_k $$

My specific confusion comes from reading Kevin Murphy's 2021 book "Probabilistic Machine Learning: An Introduction". He says something like this about Kullback-Leibler divergence (it's a paraphrase summarization of sections 4.2 and 6.2):

$KL(p||q) = \sum_{k=1}^{K} p_k \log p_k - \sum_{k=1}^{K} p_k \log q_k$

We recognize the first term as the negative entropy and the second term as the cross entropy. The first term is a constant with respect to our predictions $q$, so we can ignore it.

Let us suppose the $p$ distribution is defined with a delta function $\delta$ like this: $ p(x) = \frac{1}{N} \sum_{n=1}^{N} \delta(x - x_n)$ .

Then the KL divergence becomes \begin{align} KL(p||q) &= -H(p) - \frac{1}{N} \sum_{n=1}^{N} \log q(y_n)\\ &= constant + NLL \end{align} This is called the cross entropy objective, and is equal to the average negative log likelihood of q on the training set.


  1. The term $\frac{1}{N} \sum_{n=1}^{N} \log q(y_n)$ mentions one distribution $q$. How can it be a cross-entropy term when cross entropy is defined for two distributions $p$ and $q$?

  2. How does a log-likehood expression in terms of $N$ training instances ($\frac{1}{N} \sum_{n=1}^{N})$ turn into a cross-entropy expression in terms of $K$ classes ($\sum_{k=1}^{K}$)?

  3. Is the author's use of a delta function $\delta$ just another way of saying a one-hot distribution?

I'm still confused even after reading other posts like this one, this one, and this one.


1 Answer 1

  1. This is because of the claim about delta distributions. Now $p_k$ is $0$ for all $k$ but one. Those $p_k \log q_k$ terms in the sum over $k$ can be skipped. For the other, $p_k$ is $1$, so there’s no need to write that you multiply by $1$.
  2. I think this is answered in (1). The $0$-valued terms are skipped. In a sense, you have a double sum over $n$ and over $k$, but then the sum over $k$ goes away.
  3. Yes, that’s right. “One-hot” is terminology from the area of digital circuits; the delta function is from mathematics.
  • $\begingroup$ Thank you. For 3, I've only seen the term "one-hot" in ML books and websites. Why do you think this author would bring up delta functions in a book about machine learning? And which delta function? I know of the Dirac and Kroneker delta functions. Does it matter? $\endgroup$ May 20, 2021 at 19:56
  • 1
    $\begingroup$ Yes, ML borrowed it from circuits. Why bring up delta functions in an ML book? Because probability is mathematical, and it’s the focus of the book! Which delta depends on the data but is a longer discussion answered elsewhere on this site. To your last question: without going a rabbit hole of nuance, yes. $\endgroup$ May 20, 2021 at 20:12
  • 2
    $\begingroup$ @stackoverflowuser2010 perhaps the paper Why the logistic function? A tutorial discussion on probabilities and neural networks by Michael I. Jordan, 1995, could help. $\endgroup$
    – mhdadk
    May 20, 2021 at 20:46
  • 1
    $\begingroup$ Don’t forget the word “average” in Murphy’s description, which you quoted in the original question. It’ll fix the discrepancy. $\endgroup$ May 26, 2021 at 2:39
  • 1
    $\begingroup$ It's equivalent—try taking the log of it. $\endgroup$ Jun 1, 2021 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.