Skip to main content
2 of 5
added 7 characters in body
whuber
  • 333.6k
  • 63
  • 792
  • 1.3k

Bayesian vs Maximum entropy

Following is a question I asked in AoPS forum and I was informed that this would be a better place to ask this question. Here's the question:

Suppose that the quantity which we want to infer is a probability distribution. All we know is that the distribution comes from a set $E$ determined, say, by some of its moments and we have a prior $Q$.

The maximum entropy principle(MEP) says (supported by Sanov's theorem) that the $P^{\star}\in E$ which has least relative entropy from $Q$ (i.e., $P^{\star}=\displaystyle \text{argmin}_{P\in E}D(P\|Q)$) is the best one to select. Whereas the Bayesian rule of selection has a process of selecting the posterior given the prior which is supported by Bayes' theorem.

My question is whether there is any connection between these two inference methods (i. e., whether the two methods apply to the same problem and have something in common)? Or whether in Bayesian inference the setting is completely different from the above mentioned setting? Or am I not making sense?!

Ashok
  • 1.2k
  • 9
  • 24