Why would a cross-entropy approach negative infinity?

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

But the cross-entropy is defined as

$$−\mathbb{E}_{x∼P_{data}}\log P_{model}(y|x)$$

or in the empirical form

$$−\frac{1}{m}\sum_{i=1}^m \log P_{model}(y|x)$$

As we can see it is bounded below by $$−\log 1=0$$. Then why the book says that it would approach negative infinity?

Any thought would help!

• What you mean by "bounded below"? $P_{model}(y|x)$ is a likelihood, so it can be anything in $[0, \infty)$.
– Tim
May 15, 2021 at 8:50
• @Tim♦ I always think $P$ here is the conditional probability. Do you mean it is an unnormalized probability? That makes sense, I'll further check it! May 15, 2021 at 9:24
• First, in many cases $y$ would be a continuous random variable, so it would be a probability density rather than probability. Second, the likelihood function is defined as the unnormalized probability (density), you should not assume it is normalized.
– Tim
May 15, 2021 at 9:29
• @Tim♦ Got it, that totally makes sense! The notation in the book is a little confusing since it uses capital $P$ to define the cross-entropy and says that a capital $P$ means probability mass function.Thanks a lot, should I delete this question then? May 15, 2021 at 9:38

1 Answer

Let me bold the other part of the quote

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

When $$y$$ is a continuous, i.e. real-valued, $$P_{model}(y|x)$$ is a probability density function (e.g. Gaussian, as in the above quote), in such case it can be anything in the $$[0, \infty)$$ interval. So it is not bounded from above.

Second, likelihood function is usually unnormalized, so it does not have an upper bound and does not integrate to unity. Normalizing it is not needed for the task of finding its maximum, so the normalizing constant is often dropped. In such a case, even if the modeled variable was discrete, the likelihood would also not be bounded.