We know that the normal distribution has the maximum entropy among all continuous distribution on $\mathbb{R}$ for a given variance.
I wonder what's the opposite, i.e. what distribution has the maximum (or minimum) variance for a given entropy?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The normal distribution has the maximum entropy among all continuous distributions with fixed mean and variance on real support.
The variance for a given entropy can be made arbitrarily large using a mixture of two Gaussians that are spread farther and farther apart. The variance increases, while the entropy doesn't change much.
The minimum variance for a given entropy sounds the same to me as asking for the maximum entropy for a given variance, and so I think you're right that it is normal.
-
$\begingroup$ Thanks, I missed the "for given variance" part in the question, which has been edited. My haunch for the minimum variance is the normal distribution, but i am not sure. $\endgroup$ Commented Feb 13, 2013 at 21:57
-
$\begingroup$ The entropy of the normal distribution is an increasing function of the variance. So if for a given entropy $S$ the normal distribution has a variance $V$, then for the given entropy $S$ there exists no distribution with a lower variance than $V$ because for any distribution with smaller variance than $V$ the entropy must be smaller than $S$. $\endgroup$ Commented Jan 12, 2023 at 17:17