Is Maximum Entropy rule equivalent to non-informativeness?

In other words, when maximizing the entropy of a distribution, given some known stuff, is it equivalent to finding to most non-informative distributions? If so, how do you justify this mathematically?

  • $\begingroup$ It is as non-informative as it can mathematically be, given constraints on ignorance imposed by information. I've found John Harte's explanation in "Maximum Entropy and Ecology" to be quite accessible. $\endgroup$ Aug 16 '13 at 5:27
  • 4
    $\begingroup$ This will sound pedantic, but maximum entropy distributions are non-informative whenever you interpret entropy as "a lack of information". In many situations this interpretation makes sense, but interpretations lie outside the scope of what can be justified with mathematics. $\endgroup$
    – caburke
    Aug 16 '13 at 6:55
  • 2
    $\begingroup$ +1. For example, many people talk informally about (say) extracting the information from the data, and this often is a matter of what is scientifically or practically interesting or important and scientists and practitioners often have a good sense of that. But it is crucial here that "information" is not defined, and there are many contexts in which defining it precisely would rid us of a useful word. We could always find another word, but "informative" will not always mean "in an information theory sense" in statistical discussions. $\endgroup$
    – Nick Cox
    Aug 16 '13 at 7:53
  • $\begingroup$ Yeah, all of this make sense, but I am looking for something more formal. May be asking for too much .... $\endgroup$
    – Daniel
    Aug 16 '13 at 13:29

My understanding is that these are two ways of stating the same thing. So it's a tautology or a question of alternative terminology.

E.T. Jaynes's book is widely considered canonical. http://ksvanhorn.com/bayes/jaynes/ is a kind of back door start to the book and commentary.


The Maximum Entropy Rule is a conditioning rule for least bias in the case where the sought after distribution is under-determined. As such it gives the largest possible probability weighting to competing hypotheses given the stated constraints.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.