2
$\begingroup$

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

But the cross-entropy is defined as

$$−\mathbb{E}_{x∼P_{data}}\log P_{model}(y|x)$$

or in the empirical form

$$−\frac{1}{m}\sum_{i=1}^m \log P_{model}(y|x)$$

As we can see it is bounded below by $−\log 1=0$. Then why the book says that it would approach negative infinity?

Any thought would help!

$\endgroup$
4
  • $\begingroup$ What you mean by "bounded below"? $P_{model}(y|x)$ is a likelihood, so it can be anything in $[0, \infty)$. $\endgroup$
    – Tim
    May 15 '21 at 8:50
  • $\begingroup$ @Tim♦ I always think $P$ here is the conditional probability. Do you mean it is an unnormalized probability? That makes sense, I'll further check it! $\endgroup$
    – atlas
    May 15 '21 at 9:24
  • $\begingroup$ First, in many cases $y$ would be a continuous random variable, so it would be a probability density rather than probability. Second, the likelihood function is defined as the unnormalized probability (density), you should not assume it is normalized. $\endgroup$
    – Tim
    May 15 '21 at 9:29
  • $\begingroup$ @Tim♦ Got it, that totally makes sense! The notation in the book is a little confusing since it uses capital $P$ to define the cross-entropy and says that a capital $P$ means probability mass function.Thanks a lot, should I delete this question then? $\endgroup$
    – atlas
    May 15 '21 at 9:38
1
$\begingroup$

Let me bold the other part of the quote

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in cross-entropy approaching negative infinity.

When $y$ is a continuous, i.e. real-valued, $P_{model}(y|x)$ is a probability density function (e.g. Gaussian, as in the above quote), in such case it can be anything in the $[0, \infty)$ interval. So it is not bounded from above.

Second, likelihood function is usually unnormalized, so it does not have an upper bound and does not integrate to unity. Normalizing it is not needed for the task of finding its maximum, so the normalizing constant is often dropped. In such a case, even if the modeled variable was discrete, the likelihood would also not be bounded.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.