The Normal Distribution has maximal entropy for a given variance. Is there a distribution with minimal entropy for a given variance?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ I guess the answer should be no, since you can just have a mixture of Gaussians with different means, and send the variance of each to 0, and the standard deviation will be approximately constant, but the entropy should go to -infinity. $\endgroup$– capybaraletCommented May 12, 2017 at 1:44
-
$\begingroup$ The entropy of a normal distribution is $0.5 \text{ln}(2\pi \sigma^2) + 0.5$ and will be minimized when $\sigma = 0$. The entropy should not go to -infinity. $\endgroup$– Sextus EmpiricusCommented Jan 12, 2023 at 16:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
A Bernoulli distribution could be a candidate. But a problematic part is that the entropy is minimised when $p = 1$ or $p=0$. We can reach this state (with entropy equal to zero) as close as we like but not exactly. (Because the variance becomes zero)
More in detail: For a given variance $\sigma^2$ we would have the following distributions parameterized by $p$
$$p(x) = \begin{cases} 1-p & \text{if $x=0$}\\ p & \text{if $x=\frac{\sigma}{\sqrt{p(1-p)}}$}\\ 0 & \text{else} \end{cases}$$
And the entropy is
$$S = - (1-p) \log(1-p) - p \log(p)$$
This distribution approaches $0$ for $p \to 0$ or $p \to 1$. We can approach this limit as close as we want but the limit itself is not an existing distribution.
answered Jan 12, 2023 at 16:31