10
$\begingroup$

Suppose that the quantity which we want to infer is a probability distribution. All we know is that the distribution comes from a set $E$ determined, say, by some of its moments and we have a prior $Q$.

The maximum entropy principle(MEP) says that the $P^{\star}\in E$ which has least relative entropy from $Q$ (i.e., $P^{\star}=\displaystyle \text{argmin}_{P\in E}D(P\|Q)$) is the best one to select. Whereas the Bayesian rule of selection has a process of selecting the posterior given the prior which is supported by Bayes' theorem.

My question is whether there is any connection between these two inference methods (i. e., whether the two methods apply to the same problem and have something in common)? Or whether in Bayesian inference the setting is completely different from the above mentioned setting? Or am I not making sense?!

Improve this question
$\endgroup$
9
  • $\begingroup$ Is Q a distribution over E? $\endgroup$
    – Simon Byrne
    Commented Mar 1, 2011 at 19:01
  • $\begingroup$ Do you mean to ask, is $Q\in E$? Need not be. $\endgroup$
    – Ashok
    Commented Mar 2, 2011 at 2:24
  • 1
    $\begingroup$ You may find this question useful: stats.stackexchange.com/q/4978/495 $\endgroup$
    – Simon Byrne
    Commented Mar 3, 2011 at 11:33
  • 2
    $\begingroup$ Robin, fact is that I don't know Bayesian inference method completely. Otherwise this question would not even have arisen to me. Now I am trying to find time to learn about Bayesian. All I knew(roughly) was that using Bayes theorem if some prior and some additional information are given one can update the probabilities. I do not know this rigorously. Whereas I know MaxEnt rigorously what it means. If it is possible, please explain or lead me(i.e.,point some reference) to learn Bayesian inference rigorously. Thank you. $\endgroup$
    – Ashok
    Commented Mar 9, 2011 at 14:19
  • 1
    $\begingroup$ @Ashok most often connection you are searching for arises from the description of convex sets with a probability measure on its extreme points (Choquet Theory). $\endgroup$
    – robin girard
    Commented Mar 9, 2011 at 16:10

1 Answer 1

Reset to default
7
$\begingroup$

This may come a wee late, but the question should be rephrased: as defined by Jaynes, maximum entropy is a way to construct a prior distribution that (a) satisfies the constraints imposed by $E$ and (b) has the maximum entropy, relative to a reference measure in the continuous case: $$ \int -\log [ \pi(\theta) ] \text{d}\mu(\theta)\,. $$ Thus, (Jaynes') maximum entropy is clearly part of the Bayesian toolbox. And the maximum entropy prior does not provide the prior distribution that is closest to the true prior, as suggested by Ashok's question.

Bayesian inference about a distribution $Q$ is an altogether different problem, handled by Bayesian non-parametrics (see, e.g., this recent book by Hjort et al.). It requires to have observations from $Q$, which does not seem to be the setting of the current question...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.