For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty.

However, this conflicts with the maximum entropy principle, which states that the Normal/Gaussian distribution has maximum entropy, moreso than the just-described "greatest uncertainty" distribution. Visually though, because it is bell-shaped, the Normal distribution looks much less uncertain (more certain/concentrated) than the uniform distribution because it is concentrated around its mean.

so I'm confused, is it the uniform or the Normal distribution that exhibits the highest uncertainty/non-concentration across its density?

(Similar titled questions are more about constraints and binary data)

  • 1
    $\begingroup$ Could you tell us what you mean by the "uniform distribution"? For your question to make any sense you have to be comparing distributions supported on the entire real line, but there is no uniform distribution on the line. $\endgroup$
    – whuber
    Oct 7 '20 at 18:50

However, this conflicts with the maximum entropy principle, which states that the Normal/Gaussian distribution has maximum entropy,

You're almost there, just missed one important point:

MaxEnt does not gives you one magical distribution "that suits all", it gives you a procedure to find the "least informative" one given your constraints, that is, relevant information about your problem.

In your example, Gaussian is the MaxEnt distribution when one fixes the second moment $\langle x^2 \rangle$.

What about the Uniform distribution, then? Well, this one is given a "preferred" status from the definition of Entropy (which can be seen from Shannon's axioms, for example), so if I don't define any constraint to my MaxEnt procedure (other than a compact support), I'm going to get the Uniform distribution as the MaxEnt one.

[Extra] : what if I don't want to give the Uniform distribution this preferential status? Then one can use Relative Entropy and introduce some kind of prior distribution:

$$ S[Q|P] = - \int \mathrm{d}x\ q(x) \log \left( \frac{q(x)}{p(x)} \right) $$

Notice that if we set $p$ to be uniform (support considerations aside), we recover the expression for the Differential Entropy.

  • $\begingroup$ I read about, but don't quite get, the idea behind fixing the second moment $x^2$. Is this a natural constraint such that dropping it would be an unnatural thing to do, therefore making it always in effect empirically? I hope this detail could be explained more because it's key $\endgroup$
    – develarist
    Oct 7 '20 at 19:03
  • $\begingroup$ and how far can the corollary of maximum entropy = minimal concentration be taken? $\endgroup$
    – develarist
    Oct 7 '20 at 19:16
  • $\begingroup$ I think there are different ways to interpret it but I find it more clear to think of your constraints as things you want to model. You are the one who knows best the relevant information in your data or application, so you pick what you want to include in your model. If, for whatever reason you think fixing the variance is a good thing, then do it. The solution is going to be a Gaussian. If you think it's something else, use something else. The MaxEnt distribution will be something else then. $\endgroup$ Oct 8 '20 at 7:54
  • $\begingroup$ Not sure about minimal concentration. Where did you see this? $\endgroup$ Oct 8 '20 at 7:55
  • $\begingroup$ A zero entropy variable has maximum concentration because its one outcome makes it 100% certain. The polar opposite of this is a max entropy variable which has minimal concentration because its probability distribution is spread out across several different outcomes. Do you get it now? $\endgroup$
    – develarist
    Oct 8 '20 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.