How does a distribution's differential entropy correspond to its moments?

Question

The Gaussian distribution maximizes entropy for the following functional constraints

$$E(x) = \mu$$ and $$E((x-\mu)^2) = \sigma^2$$

which are just its first and second statistical moments (true parameters, not estimates of them),

as well the constraint that $x$ be included in the support of the probability density, which for the Gaussian is $(-\infty, \infty)$.

Is the above suggesting some sort of link between entropy and moments? By imposing those constraints (knowing the true moments?), we can be assured of maximum entropy as well as our estimated entropy value? Does this suggest that the statistical moments and entropy can be defined by one another and that, if I have the moments, I can calculate the corresponding entropy and vice versa? This would contradict the fact that several distributions with differing moments can have identical entropies though

Source table of distributions, their constraints and supports that provide closed-form analytical maximum entropy solutions. scroll to Other Examples for the table

Answer 1

I would not expect any near relationship between differential entropy and moments, excepting maybe special cases. Differential entropy of the random variable $X$ with density $f(x)$ is $$ H=-\int f(x) \log f(x) \; dx $$ note the possible values $x$ does appear only as an argument to the density, not alone, while for the moments $$\DeclareMathOperator{\E}{\mathbb{E}} \E X^p = \int x^p f(x)\; dx $$ it appears alone.

If you try to expand in the integral for differential entropy $\log f(x)$ in a power series, you will get a power series in $f(x)$, not in $x$. So I cannot see from where an analytical relationship should appear.

But $f(X)$ is itself a random variable, and the above indicates some relationship to the moments of $f(X)$. And in many cases, particularly the normal distribution, and, more generally, exponential families, $\log f(x)$ will have a simple expression in $x$. For the standard normal, $$\log \phi(x) = \text{constant}-\frac{x^2}2 $$ so the differential entropy will be a simple function of the variance. But outside exponential families we cannot expect such simplifications. And, behold, most examples of maxent distributions are exponential families ...

${}$

Answer 2

If there was functional relation (a "formula") between the moments of a distribution and its differential entropy, then any two distributions with identical moments would have identical differential entropies. This is not the case. I can demonstrate that @kjetil's intuition is correct with an example of two distributions with identical moments that have different differential entropies.

It is well known that the following class of "perturbed log-normal" distributions have identical moments, despite having different pdfs:

$$ f_a (x) = \frac{1}{x \sqrt{2 \pi}} \exp \left( \frac{-1}{2} [\log(x)]^2 \right)(1 + a \sin(2 \pi \log (x))). $$ for $0 \le a \le 1$. When $a = 0$, this is simply the distribution $\text{Lognormal} (0, 1)$, which has the well-known entropy $h_0 = -\int \log(f_0(x))f_0(x) dx = \log(\sqrt{2\pi e}) \approx 1.41894$. For other values of $a$ the integral defining entropy seems intractable (Mathematica couldn't simplify it); however, calculating the entropy by integrating numerically (I used Mathematica's NIntegrate function with a PrecisionGoal of 5 digits) yields $h_1 = 1.11209$, which is not equal to $h_0 = 1.41894$. This shows that the moments cannot completely determine the differential entropy, as otherwise the entropies $h_0$ and $h_1$ would be identical.