Why would a cross-entropy approach negative infinity?

Question

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

But the cross-entropy is defined as

$$−\mathbb{E}_{x∼P_{data}}\log P_{model}(y|x)$$

or in the empirical form

$$−\frac{1}{m}\sum_{i=1}^m \log P_{model}(y|x)$$

As we can see it is bounded below by $−\log 1=0$. Then why the book says that it would approach negative infinity?

Any thought would help!

Answer 1

Let me bold the other part of the quote

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

When $y$ is a continuous, i.e. real-valued, $P_{model}(y|x)$ is a probability density function (e.g. Gaussian, as in the above quote), in such case it can be anything in the $[0, \infty)$ interval. So it is not bounded from above.

Second, likelihood function is usually unnormalized, so it does not have an upper bound and does not integrate to unity. Normalizing it is not needed for the task of finding its maximum, so the normalizing constant is often dropped. In such a case, even if the modeled variable was discrete, the likelihood would also not be bounded.