# Why would a cross-entropy approach negative infinity?

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

But the cross-entropy is defined as

$$−\mathbb{E}_{x∼P_{data}}\log P_{model}(y|x)$$

or in the empirical form

$$−\frac{1}{m}\sum_{i=1}^m \log P_{model}(y|x)$$

As we can see it is bounded below by $$−\log 1=0$$. Then why the book says that it would approach negative infinity?

Any thought would help!

• What you mean by "bounded below"? $P_{model}(y|x)$ is a likelihood, so it can be anything in $[0, \infty)$.
– Tim
May 15, 2021 at 8:50
• @Tim♦ I always think $P$ here is the conditional probability. Do you mean it is an unnormalized probability? That makes sense, I'll further check it! May 15, 2021 at 9:24
• First, in many cases $y$ would be a continuous random variable, so it would be a probability density rather than probability. Second, the likelihood function is defined as the unnormalized probability (density), you should not assume it is normalized.
– Tim
May 15, 2021 at 9:29
• @Tim♦ Got it, that totally makes sense! The notation in the book is a little confusing since it uses capital $P$ to define the cross-entropy and says that a capital $P$ means probability mass function.Thanks a lot, should I delete this question then? May 15, 2021 at 9:38

When $$y$$ is a continuous, i.e. real-valued, $$P_{model}(y|x)$$ is a probability density function (e.g. Gaussian, as in the above quote), in such case it can be anything in the $$[0, \infty)$$ interval. So it is not bounded from above.