# Statistical interpretation of Maximum Entropy Distribution

I have used the principle of maximum entropy to justify the use of several distributions in various settings; however, I have yet to be able to formulate a statistical, as opposed to information-theoretic, interpretation of maximum entropy. In other words, what does maximizing the entropy imply about the statistical properties of the distribution?

Has anyone run across or perhaps discovered yourself a statistical interpretation of max. entropy distributions that does not appeal to information, but only to probabilistic concepts?

As an example of such an interpretation (not necessarily true): "For an interval of arbitrary length L on the domain of the RV (assuming its 1-d continuous for simplicity), the maximum probability that can be contained in this interval is minimized by the maximum entropy distribution."

So, you see there is no talk about "informativeness" or other more philosophical ideas, just probabilistic implications.

• I think you have to be more specific about what you are looking for: entropy is after all as "statistical" a measure as variance etc. so the maximum entropy distribution maximises entropy is a perfectly good statistical description. So it seems to me you have to go outside statistics to come up with a "justification" Aug 1 '13 at 9:48
• Seanv: I agree that entropy, as a statistical functional, is just as "statistical" as variance, expected value, skew etc. However, using mean and standard deviation as examples, these have purely probabilistic interpretations via Markov's and Chebyshev's theorems and ultimately in one of a number of central limit theorems and also intuitively as long run sums (for the mean) and RMS error (for the standard deviation). I should perhaps repharase my question to read "Probabilistic interpretation of maximum entropy distributions". Aug 1 '13 at 12:24
• Annika, maximum entropy distribution has the following interpretation: If $X_1,X_2,\dots$ are i.i.d. random variables, then the conditional probalitity $P(\cdot|X_1+\dots+X_n=na)\to P^*(\cdot)$ as $n\to \infty$ where $P^*$ is the maximum entropy distribution from the set $\{P:\mathbb{E}_PX=a\}$. See also ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1056374&tag=1 Aug 1 '13 at 12:45
• Thanks Ashok. Ill take a look at that paper in more detail. This seems like a specific case of maximizing entropy for a given mean, but I am still curious as to what the operation of maximizing the Shanon entropy is doing mathematically such that the above result holds? Is it effectively minimizing the maximum density or average concentration of the probability measure? Aug 1 '13 at 15:55

This isn't really my field, so some musings:

I will start with the concept of surprise. What does it mean to be surprised? Usually, it means that something happened that was not expected to happen. So, surprise it a probabilistic concept and can be explicated as such (I J Good has written about that). See also Wikipedia and Bayesian Surprise.

Take the particular case of a yes/no situation, something can happen or not. It happens with probability $$p$$. Say, if p=0.9 and it happens, you are not really surprised. If $$p=0.05$$ and it happens, you are somewhat surprised. And if $$p=0.0000001$$ and it happens, you are really surprised. So, a natural measure of "surprise value in observed outcome" is some (anti)monotone function of the probability of what happened. It seems natural (and works well ...) to take the logarithm of probability of what happened, and then we throw in a minus sign to get a positive number. Also, by taking the logarithm we concentrate on the order of the surprise, and, in practice, probabilities are often only known up to order, more or less.

So, we define $$\text{Surprise}(A) = -\log p(A)$$ where $$A$$ is the observed outcome, and $$p(A)$$ is its probability.

Now we can ask what is the expected surprise. Let $$X$$ be a Bernoulli random variable with probability $$p$$. It has two possibly outcomes, 0 and 1. The respective surprise values is \begin{align} \text{Surprise}(0) &= -\log(1-p) \\ \text{Surprise}(1) &= -\log p \end{align} so the surprise when observing $$X$$ is itself a random variable with expectation $$p \cdot -\log p + (1-p) \cdot -\log(1-p)$$ and that is --- surprise! --- the entropy of $$X$$! So entropy is expected surprise!

Now, this question is about maximum entropy. Why would anybody want to use a maximum entropy distribution? Well, it must be because they want to be maximally surprised! Why would anybody want that?

A way to look at it is the following: You want to learn about something, and to that goal you set up some learning experiences (or experiments ...). If you already knew everything about this topic, you are able to always predict perfectly, so are never surprised. Then you never get new experience, so do not learn anything new (but you know everything already---there is nothing to learn, so that is OK). In the more typical situation that you are confused, not able to predict perfectly, there is a learning opportunity! This leads to the idea that we can measure the "amount of possible learning" by the expected surprise, that is, entropy. So, maximizing entropy is nothing other than maximizing opportunity for learning. That sounds like a useful concept, which could be useful in designing experiments and such things.

A poetic example is the well known

Wenn einer eine reise macht, dann kann er was erzählen ...

One practical example: You want to design a system for online tests (online meaning that not everybody gets the same questions, the questions are chosen dynamically depending on previous answers, so optimized, in some way, for each person).

If you make too difficult questions, so they are never mastered, you learn nothing. That indicates you must lower the difficulty level. What is the optimal difficulty level, that is, the difficulty level which maximizes the rate of learning? Let the probability of correct answer be $$p$$. We want the value of $$p$$ that maximizes the Bernoulli entropy. But that is $$p=0.5$$. So you aim to state questions where the probability of obtaining a correct answer (from that person) is 0.5.

Then the case of a continuous random variable $$X$$. How can we be surprised by observing $$X$$? The probability of any particular outcome $$\{X=x\}$$ is zero, the $$-\log p$$ definition is useless. But we will be surprised if the probability of observing something like $$x$$ is small, that is, if the density function value $$f(x)$$ is small (assuming $$f$$ is continuous). That leads to the definition $$\DeclareMathOperator{\E}{\mathbb{E}} \text{Surprise}(x) = -\log f(x)$$ With that definition, the expected surprise from observing $$X$$ is $$\E \{-\log f(X)\} = -\int f(x) \log f(x) \; dx$$ that is, the expected surprise from observing $$X$$ is the differential entropy of $$X$$. It can also be seen as the expected negative loglikelihood.

But this isn't really the same as the first, event, case. Too see that, an example. Let the random variable $$X$$ represent the length of a throw of a stone (say in a sports competition). To measure that length we need to choose a length unit, since there is no intrinsic scale to length, as there is to probability. We could measure in mm or in km, or more usually, in meters. But our definition of surprise, hence expected surprise, depends on the unit chosen, so there is no invariance. For that reason, the values of differential entropy are not directly comparable the way that Shannon entropy is. It might still be useful, if one remembers this problem.

• This is one of the best and intuitive explanations of maximum entropy that I've seen! Jan 10 '18 at 21:50
• This one was a nice explanation. Thank you. Oct 8 '20 at 14:14

Perhaps not exactly what you are after, but in Rissanen, J. Stochastic Complexity in Statistical Inquiry, World Scientific, 1989, p. 41 there is an interesting connection of maximum entropy, the normal distribution and the central limit theorem. Among all densities with mean zero and standard deviation $\sigma$, the normal density has maximum entropy.

"Hence, in this interpretation the basic central limit theorem expresses the fact that the per symbol entropy of sums of independent random variables with mean zero and common variance tends to the maximum. This seems eminently reasonable; in fact, it is an expression of the second law of thermodynamics, which Eddington viewed as holding 'the supreme position among the laws of Nature'."

I have not yet explored the implications of this, nor am I sure I fully understand them.

[edit: fixed typo]

While not an expert in information theory and maximum entropy, I've been interested in it for a while.

The entropy is a measure of the uncertainty of a probability distribution that was derived according to a set of criteria. It and related measures characterize probability distributions. And, it's the unique measure that satisfies those criteria. This is similar to the case of probability itself, which as explained beautifully in Jaynes (2003), is the unique measure that satisfies some very desirable criteria for any measure of uncertainty of logical statements.

Any other measure of the uncertainty of a probability distribution that was different than entropy would have to violate one or more of the criteria used to define entropy (otherwise it would necessarily be entropy). So, if you had some general statement in terms of probability that somehow gave the same results as maximum entropy... then it would be maximum entropy!

The closest thing I can find to a probability statement about maximum entropy distributions so far is Jaynes's concentration theorem. You can find it clearly explained in Kapur and Kesavan (1992). Here is a loose restatement:

We require a discrete probability distribution $p$ on $n$ outcomes. That is, we require $p_i$, $i=1,...,n$. We have $m$ constraints that our probability distribution has to satisfy; additionally, since probabilities must add to 1 we have a total of $m+1$ constraints.

Let $S$ be the entropy of some distribution that satisfies the $m+1$ constraints and let $S_{\textrm{max}}$ be the entropy of the maximum entropy distribution.

As the size of the set of observations $N$ grows, we have $$2N(S_{\textrm{max}} - S) \sim \chi^2_{n-m-1}.$$

With this, a 95% entropy interval is defined as $$\left( S_{\textrm{max}} - \frac {\chi^2_{n-m-1} (0.95)}{2N}, S_{\textrm{max}} \right).$$ So, any other distribution that satisfies the same constraints as the maximum entropy distribution has a 95% chance of having entropy greater than $S_{\textrm{max}} - \frac {\chi^2_{n-m-1} (0.95)}{2N}$.

E.T. Jaynes (2003) Probability Theory: The Logic of Science. Cambridge University Press.

J.N. Kapur and .K. Kesavan (1992) Entropy Optimization Principles with Applications. Academic Press, Inc.

You might want to have a look at the Wallis derivation.

https://en.wikipedia.org/wiki/Principle_of_maximum_entropy#The_Wallis_derivation

It has the advantage of being strictly combinatorial in nature, making no reference to information entropy as a measure of 'uncertainty', 'uninformativeness', or any other imprecisely defined concept.

Suppose you have a dice. If the dice is fair, the average value of the number shown will be 3.5. Now, imagine to have a dice for which the average value shown is a bit higher, let's say 4.

How can it do that? Well, it could do it in zillion ways! It could for example show 4 every single time. Or it could show 3, 4, 5 with equal probability.

Let's say you want to write a computer program that simulates a dice with average 4. How would you do it?

An interesting solution is this. You start with a fair dice. You roll it many times (say 100) and you get a bunch of numbers. If the average of these numbers is 4, you accept the sample. Otherwise you reject it and try again.

After many many attempts, you finally get a sample with average 4. Now your computer program will simply return a number randomly chosen from this sample.

Which numbers will it show? Well, for example, you expect 1 to be present a little bit, but probably not 1/6 of the times, because a 1 will lower that average of the sample and it will increase the probability of the sample to be rejected.

In the limit of a very big sample, the numbers will be distributed according to this:

https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#Discrete_distributions_with_specified_mean

which is the distribution with maximum entropy among the ones with specified mean. Aha!