3
$\begingroup$

I'm reading Bishop's Pattern Recognition and Machine Learning. In chapter 1.6: Information Theory (page 53) when trying to derive the maximum differential entropy pdf from the definition of continuous entropy, the author states:

"Let us now consider the maximum entropy configuration for a continuous variable. In order for this maximum to be well defined, it will be necessary to constrain the first and second moments of p(x) as well as preserving the normalization constraint" (where p(x) is the pdf we wish to learn the form of)

My two questions:

  1. What does it mean for a maximum to be well-defined? Intuition + math please!
  2. Why does constraining the first and second moments of p(x) ensure this maximum is well defined - i.e., why do we not need to constrained all moments.

Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ It is instructive to find a maximum entropy distribution subject to a constraint only on its mean--or even subject to no constraints at all. What would it look like? $\endgroup$
    – whuber ♦
    Sep 24, 2014 at 21:20
  • $\begingroup$ Hm - i suspect you are hinting that the resulting pdf would just be an inproper distribution (so a horizontal line if x is one dimensional)? However surely this could be avoided by constraining integral over p(x) = 1? For example, cauchy distribution has an undefined mean if I recall correctly? $\endgroup$
    – user3279453
    Sep 25, 2014 at 7:02

1 Answer 1

Reset to default
3
$\begingroup$

  1. The maximum is "well-defined" iff there exists a probability density meeting the constraints whose entropy is not smaller than any other probability density meeting the constraints. If you do not apply any constraints (other than normalization and non-negativity) then the maximum entropy distribution is not well-defined, since for any probability density you pick there is another density whose entropy is larger (for example, a Gaussian with sufficiently large variance).

  2. The book shows that constraining the first and second moments are sufficient to obtain a well-defined maximum.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.