1
$\begingroup$

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood".

To start, I already fully understand these definitions:

  1. Entropy of a probability distribution $p$ with $K$ classes:

$$ H(p) = - \sum_{k=1}^{K} p_k \log p_k $$

  1. Cross entropy between two probability distributions $p$ (ground-truth) and $q$ (predicted):

$$ H(p, q) = - \sum_{k=1}^{K} p_k \log q_k $$

My specific confusion comes from reading Kevin Murphy's 2021 book "Probabilistic Machine Learning: An Introduction". He says something like this about Kullback-Leibler divergence (it's a paraphrase summarization of sections 4.2 and 6.2):

$KL(p||q) = \sum_{k=1}^{K} p_k \log p_k - \sum_{k=1}^{K} p_k \log q_k$

We recognize the first term as the negative entropy and the second term as the cross entropy. The first term is a constant with respect to our predictions $q$, so we can ignore it.

Let us suppose the $p$ distribution is defined with a delta function $\delta$ like this: $ p(x) = \frac{1}{N} \sum_{n=1}^{N} \delta(x - x_n)$ .

Then the KL divergence becomes \begin{align} KL(p||q) &= -H(p) - \frac{1}{N} \sum_{n=1}^{N} \log q(y_n)\\ &= constant + NLL \end{align} This is called the cross entropy objective, and is equal to the average negative log likelihood of q on the training set.

Questions:

  1. The term $\frac{1}{N} \sum_{n=1}^{N} \log q(y_n)$ mentions one distribution $q$. How can it be a cross-entropy term when cross entropy is defined for two distributions $p$ and $q$?

  2. How does a log-likehood expression in terms of $N$ training instances ($\frac{1}{N} \sum_{n=1}^{N})$ turn into a cross-entropy expression in terms of $K$ classes ($\sum_{k=1}^{K}$)?

  3. Is the author's use of a delta function $\delta$ just another way of saying a one-hot distribution?

I'm still confused even after reading other posts like this one, this one, and this one.

$\endgroup$
2
$\begingroup$
  1. This is because of the claim about delta distributions. Now $p_k$ is $0$ for all $k$ but one. Those $p_k \log q_k$ terms in the sum over $k$ can be skipped. For the other, $p_k$ is $1$, so there’s no need to write that you multiply by $1$.
  2. I think this is answered in (1). The $0$-valued terms are skipped. In a sense, you have a double sum over $n$ and over $k$, but then the sum over $k$ goes away.
  3. Yes, that’s right. “One-hot” is terminology from the area of digital circuits; the delta function is from mathematics.
$\endgroup$
12
  • $\begingroup$ Thank you. For 3, I've only seen the term "one-hot" in ML books and websites. Why do you think this author would bring up delta functions in a book about machine learning? And which delta function? I know of the Dirac and Kroneker delta functions. Does it matter? $\endgroup$ – stackoverflowuser2010 May 20 at 19:56
  • $\begingroup$ For 1, I know that to compute total cross-entropy loss over a data set, you compute the mean cross-entropy loss over each training instances, like $H_{total}(P, Q) = -\frac{1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} p_{k,n} \log q_{k,n}$ for class $k$ and training instance $n$. So the relation between cross-entropy and negative log likelihood is that the true distribution $p$ is implicitly one-hot, so you can remove the summation over $K$ ? Is that all there is??? $\endgroup$ – stackoverflowuser2010 May 20 at 20:03
  • $\begingroup$ Yes, ML borrowed it from circuits. Why bring up delta functions in an ML book? Because probability is mathematical, and it’s the focus of the book! Which delta depends on the data but is a longer discussion answered elsewhere on this site. To your last question: without going a rabbit hole of nuance, yes. $\endgroup$ – Arya McCarthy May 20 at 20:12
  • $\begingroup$ For 1, can you please point me to an understandable reference that explains whatever nuance I'm missing? $\endgroup$ – stackoverflowuser2010 May 20 at 20:26
  • 2
    $\begingroup$ @stackoverflowuser2010 perhaps the paper Why the logistic function? A tutorial discussion on probabilities and neural networks by Michael I. Jordan, 1995, could help. $\endgroup$ – mhdadk May 20 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.