Comparison between MaxEnt, ML, Bayes and other kind of statistical inference methods I'm in no way a statistician (I've had a course in mathematical statistics but nothing more than that), and recently, while studying information theory and statistical mechanics, I met this thing called "uncertainty measure"/"entropy". I read Khinchin derivation of it as a measure of uncertainty and it made sense to me. Another thing that made sense was Jaynes description of MaxEnt to get a statistic when you know the arithmetic mean of one or more function/s on the sample (assuming you accept $-\sum p_i\ln p_i$ as a measure of uncertainty of course).  
So I searched on the net to find the relationship with other methods of statistical inference, and God was I confused. For example this paper suggest, assuming that i got it right, that you just get a ML estimator under a suitable reformulation of the problem; MacKey, in his book, says that MaxEnt can give you weird things, and you should't use it even for a starting estimate in a Bayesian inference; etc.. I'm having trouble finding good comparisons.
My question is, could you provide an explanation and/or good refences of weak and strong points of MaxEnt as a statistical inference method with quantitative comparisons to other methods (when applied to toy models for example)? 
 A: It is true that in the past, MaxEnt and Bayes have dealt with different types or forms of information. I would say that Bayes uses "hard" constraints as well though, the likelihood.
In any case, it is not an issue anymore as Bayes Rule (not the product rule) can be obtained from Maximum relative Entropy (MrE), and not in an ambiguous way:


*

*Updating Probabilities with Data and Moments

*From Physics to Economics: An Econometric Example Using Maximum Relative Entropy
It's a new world...
A: For an entertaining critique of maximum entropy methods, I'd recommend reading some old newsgroup posts on sci.stat.math and sci.stat.consult, particularly the ones by Radford Neal:


*

*How informative is the Maximum Entropy method? (1994)

*Maximum Entropy Imputation  (2002)

*Explanation of Maximum Entropy  (2004)
I'm not aware of any comparisons between maxent and other methods: part of the problem seems to be that maxent is not really a framework, but an ambiguous directive ("when faced with an unknown, simply maximise the entropy"), which is interpreted in different ways by different people.
A: MaxEnt and Bayesian inference methods correspond to different ways of incorporating information into your modeling procedure. Both can be put on axiomatic ground (John Skilling's "Axioms of Maximum Entropy" and Cox's "Algebra of Probable Inference").
Bayesian approach is straightforward to apply if your prior knowledge comes in a form of a measurable real-valued function over your hypothesis space, so called "prior". MaxEnt is straightforward when the information comes as a set of hard constraints on your hypothesis space. In real life, knowledge comes neither in "prior" form nor in "constraint" form, so success of your method depends on your ability to represent your knowledge in the corresponding form. 
On a toy problem, Bayesian model averaging will give you lowest average log-loss (averaged over many model draws) when the prior matches the true distribution of hypotheses. MaxEnt approach will give you lowest worst-case log-loss when its constraints are satisfied (worst taken over all possible priors)
E.T.Jaynes, considered a father of "MaxEnt" methods also relied on Bayesian methods. On page 1412 of his book, he gives an example where Bayesian approach resulted in a good solution, followed by an example where MaxEnt approach is more natural.
Maximum likelihood essentially takes the model to lie inside some pre-determined model space and trying to fit it "as hard as possible" in a sense that it'll have the highest sensitivity to data out of all model-picking methods restricted to such model space. Whereas MaxEnt and Bayesian are frameworks, ML is a concrete model fitting method, and for some particular design choices, ML can end up the method coming out of Bayesian or MaxEnt approach. For instance, MaxEnt with equality constraints is equivalent to Maximum Likelihood fitting of a certain exponential family. Similarly, an approximation to Bayesian Inference can lead to regularized Maximum Likelihood solution. If you choose your prior to make your conclusions maximally sensitive to data, result of Bayesian inference will correspond to Maximum Likelihood fitting. For instance, when inferring $p$ over Bernoulli trials, such prior would be the limiting distribution Beta(0,0)
Real-life Machine Learning successes are often a mix of various philosophies. For instance, "Random Fields" were derived from MaxEnt principles. Most popular implementation of the idea, regularized CRF, involves adding a "prior" on the parameters. As a result, the method is not really MaxEnt nor Bayesian, but  influenced by both schools of thought.
I've collected some links on philosophical foundations of Bayesian and MaxEnt approaches here and here.
Note on terminology: sometimes people call their method Bayesian simply if it uses Bayes rule at some point. Likewise, "MaxEnt" is sometimes used for some method that favors high entropy solutions. This is not the same as "MaxEnt inference" or "Bayesian inference" as described above
