I'm in no way a statistician (I've had a course in mathematical statistics but nothing more than that), and recently, while studying information theory and statistical mechanics, I met this thing called "uncertainty measure"/"entropy". I read Khinchin derivation of it as a measure of uncertainty and it made sense to me. Another thing that made sense was Jaynes description of MaxEnt to get a statistic when you know the arithmetic mean of one or more function/s on the sample (assuming you accept $-\sum p_i\ln p_i$ as a measure of uncertainty of course).

So I searched on the net to find the relationship with other methods of statistical inference, and God was I confused. For example this paper suggest, assuming that i got it right, that you just get a ML estimator under a suitable reformulation of the problem; MacKey, in his book, says that MaxEnt can give you weird things, and you should't use it even for a starting estimate in a Bayesian inference; etc.. I'm having trouble finding good comparisons.

My question is, could you provide an explanation and/or good refences of weak and strong points of MaxEnt as a statistical inference method with quantitative comparisons to other methods (when applied to toy models for example)?


MaxEnt and Bayesian inference methods correspond to different ways of incorporating information into your modeling procedure. Both can be put on axiomatic ground (John Skilling's "Axioms of Maximum Entropy" and Cox's "Algebra of Probable Inference").

Bayesian approach is straightforward to apply if your prior knowledge comes in a form of a measurable real-valued function over your hypothesis space, so called "prior". MaxEnt is straightforward when the information comes as a set of hard constraints on your hypothesis space. In real life, knowledge comes neither in "prior" form nor in "constraint" form, so success of your method depends on your ability to represent your knowledge in the corresponding form.

On a toy problem, Bayesian model averaging will give you lowest average log-loss (averaged over many model draws) when the prior matches the true distribution of hypotheses. MaxEnt approach will give you lowest worst-case log-loss when its constraints are satisfied (worst taken over all possible priors)

E.T.Jaynes, considered a father of "MaxEnt" methods also relied on Bayesian methods. On page 1412 of his book, he gives an example where Bayesian approach resulted in a good solution, followed by an example where MaxEnt approach is more natural.

Maximum likelihood essentially takes the model to lie inside some pre-determined model space and trying to fit it "as hard as possible" in a sense that it'll have the highest sensitivity to data out of all model-picking methods restricted to such model space. Whereas MaxEnt and Bayesian are frameworks, ML is a concrete model fitting method, and for some particular design choices, ML can end up the method coming out of Bayesian or MaxEnt approach. For instance, MaxEnt with equality constraints is equivalent to Maximum Likelihood fitting of a certain exponential family. Similarly, an approximation to Bayesian Inference can lead to regularized Maximum Likelihood solution. If you choose your prior to make your conclusions maximally sensitive to data, result of Bayesian inference will correspond to Maximum Likelihood fitting. For instance, when inferring $p$ over Bernoulli trials, such prior would be the limiting distribution Beta(0,0)

Real-life Machine Learning successes are often a mix of various philosophies. For instance, "Random Fields" were derived from MaxEnt principles. Most popular implementation of the idea, regularized CRF, involves adding a "prior" on the parameters. As a result, the method is not really MaxEnt nor Bayesian, but influenced by both schools of thought.

I've collected some links on philosophical foundations of Bayesian and MaxEnt approaches here and here.

Note on terminology: sometimes people call their method Bayesian simply if it uses Bayes rule at some point. Likewise, "MaxEnt" is sometimes used for some method that favors high entropy solutions. This is not the same as "MaxEnt inference" or "Bayesian inference" as described above

  • $\begingroup$ Thanks. I didn't think that "The logic of science" talked about this stuff too, i'm definitely going to read that book. $\endgroup$ – Francesco Nov 29 '10 at 17:15

For an entertaining critique of maximum entropy methods, I'd recommend reading some old newsgroup posts on sci.stat.math and sci.stat.consult, particularly the ones by Radford Neal:

I'm not aware of any comparisons between maxent and other methods: part of the problem seems to be that maxent is not really a framework, but an ambiguous directive ("when faced with an unknown, simply maximise the entropy"), which is interpreted in different ways by different people.

  • 3
    $\begingroup$ (+1) That 2002 thread is a hell of an exchange of ideas. $\endgroup$ – whuber Nov 28 '10 at 22:50
  • $\begingroup$ Note that the "wallis derivation" of maxent given by Edwin Jaynes in Probability Theory: The Logic of Science does give an "experimental" rationale for maximising entropy. In discrete distributions, if we start from principle of indifference (PID), and then basically perform rejection sampling on the probabilities, using the constraints to accept or reject the random uniform samples. The resulting probability is then arbitrarily close to the (discrete) maxent distribution. $\endgroup$ – probabilityislogic Mar 12 '12 at 12:23

It is true that in the past, MaxEnt and Bayes have dealt with different types or forms of information. I would say that Bayes uses "hard" constraints as well though, the likelihood.

In any case, it is not an issue anymore as Bayes Rule (not the product rule) can be obtained from Maximum relative Entropy (MrE), and not in an ambiguous way:

It's a new world...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.