19
$\begingroup$

What is the maximum entropy distribution for a positive continuous variable, given its first and second moments?

For example, a Gaussian distribution is the maximum entropy distribution for an unbounded variable, given its mean and standard deviation, and a Gamma distribution is the maximum entropy distribution for a positive variable, given its mean value and the mean value of its logarithm.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
16
$\begingroup$

One may simply use the theorem of Boltzmann's that is in the very Wikipedia article you point to.

Note that specifying the mean and variance is equivalent to specifying the first two raw moments - each determines the other (it's not actually necessary to invoke this, since we may apply the theorem directly to the mean and variance, it's just a little simpler this way).

The theorem then establishes that the density must be of the form:

$$f(x)=c \exp\left(\lambda_1 x + \lambda_2 x^2 \right)\quad \mbox{ for all } x \geq 0$$

Integrability over the positive real line will restrict $\lambda_2$ to be $\leq 0$, and I think places some restrictions on the relationships between the $\lambda$s (which will presumably be satisfied automatically when starting from the specified mean and variance rather than the raw moments).

To my surprise (since I wouldn't have expected it when I started this answer), this appears to leave us with a truncated normal distribution.

As it happens, I don't think I've used this theorem before, so criticisms or helpful suggestions on anything I haven't considered or have left out would be welcome.

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 Thanks. Seems alright. When I read the Wikipedia article, I seem to have missed the fact that Boltzmann theorem applies for all closed intervals. I had assumed it applied only to variables going from $-\infty$ to $\infty$. $\endgroup$
    – a06e
    Commented Jan 23, 2014 at 0:55
  • 1
    $\begingroup$ For some reason the uniform base measure and resulting truncated normal distribution do not fully convince me: As Fred Schoen stresses, to find the maximum-(relative-)entropy in the continuous case we need a base measure or reference probability distribution. Since the continuous variable $x$ in question is positive, it could be a scale variable, and a base measure proportional to $1/x$ then recommends itself for various reasons (e.g. group invariance; see Jaynes's book or Jeffreys's). $\endgroup$
    – pglpm
    Commented Jul 28, 2017 at 11:20
  • 1
    $\begingroup$ With this base measure the resulting distribution is proportional to $$ \frac{1}{x}\exp(-\alpha x- \beta x^2) $$ but unfortunately it's non-normalizable (it could still be used as an improper prior, though). Given the positiveness of the variable in question it may be worth considering whether the moments of its logarithm may make more sense as information carriers and maximum-entropy constraints. They would lead to gamma-like maximum-entropy distributions. $\endgroup$
    – pglpm
    Commented Jul 28, 2017 at 11:20
8
$\begingroup$

I want to make @Glen_b's answer more explicit, here is an extra answer just because it wouldn't fit as a comment.

The formalism etc. is well explained in Chapter 11 and 12 of Jaynes' book. Taking the uniform distribution as the base measure, the general solution, as @Glen_b already said, is a Gaussian $$ f(x) \propto \mathcal{N}(x | -1/2 \lambda_1/\lambda_2, -1/(2\lambda_2)) $$ For the unbounded variable, you can explicitly solve for the Lagrange multipliers $\lambda_1$ and $\lambda_2$ in terms of the constraint values ($a_1, a_2$ in the Wikipedia article). With $a_1=\mu, a_2=\mu^2 + \sigma^2$, you then get $\lambda_1=\mu/\sigma^2, \lambda_2=-0.5 \sigma^2$, so the standard Gaussian $\mathcal{N}(x|\mu, \sigma^2)$.

For the bounded variable $x>x_{min}$, I (and mathematica) cannot solve for $\lambda_{1,2}$ explicitly anymore because of the error function term that appears when computing the partition function ($1/c$ in wikipedia). This means that that the $\mu$ and $\sigma^2$ parameters of the truncated Gaussian are not the mean and variance of the continuous variable you started with. It can even happen that for $x_{min}=0$, the mode of the Gaussian is negative! Of course the numbers all agree again when you take $x_{min} \to -\infty$.

If you have concrete values for $a_1, a_2$, you can still solve for $\lambda_{1,2}$ numerically and plug in the solutions into the general equation and you are done! The values of $\lambda_{1,2}$ from the unbounded case may be a good starting point for the numerical solver.

This question is a duplicate of https://math.stackexchange.com/questions/598608/what-is-the-maximum-entropy-distribution-for-a-continuous-random-variable-on-0

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.