1
$\begingroup$

Suppose I have a random variable X which can take integer values 0 through 99. It can be, for example, numbers written on balls in a large urn. I have three hypotheses about the distribution of these numbers on the balls: $H_0$ says each number is just as likely as every other number; $H_1$ says that there's a 90% chance that the number on any given ball is between 50 and 59 including; and $H_2$ says that there's a 90% chance that the number on any given ball is 55.

I want to know, based on the Principle of Maximum Entropy, what the priors for each of these hypotheses should be, and how to calculate that. It feels intuitive to me that $P(H_1):P(H_2)$ should equal $10:1$ or something like that but I don't know how to show that.

$\endgroup$
1
$\begingroup$

Technically, if you want to use the "maximum" entropy principle, then in this case you would simply choose the hypothesis with the maximum entropy, which would be the $H_0$, the uniform distribution.

An alternate approach, motivated by statistical thermodynamics, would be to weight the priors according to their exponentiated entropy, so $\mathcal P[H_i]\propto e^{\mathcal H(H_i)}$, where $$\mathcal H(H_i)=-\sum_{j=1}^{100} p_{H_i}(j)\log p_{H_i}(j).$$

It is easy to compute: $$\begin{align} \mathcal H(H_0) &= \log 100 = 4.60517\\ \mathcal H(H_0) &= \log 900 -\frac{810\log 81 + 90\log 1}{900} = 2.847391\\ \mathcal H(H_0) &= \log 990 - \frac{891\log 891 + 99\log 1}{990} = 0.784595\\ \end{align}$$

Exponentiating and normalizing yields $\mathcal P(H_0) = 0.83728$, $\mathcal P(H_1) = 0.14437$, $\mathcal P(H_2) = 0.01835$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Hmm... isn't MAXENT supposed to be used to find out what our priors ought to be in the first place? I'm fuzzy on this part, sorry. So my priors depend specifically on the hypotheses I have considered and don't include cases for all the hypotheses I haven't? $\endgroup$ – Pedro Carvalho Oct 6 '13 at 21:46
  • $\begingroup$ I don't believe that version of the maximum entropy principle is really applicable here. Selecting a prior based on maximum entropy generally means selecting a distribution which maximizes entropy from the set of all distributions satisfying certain constraints. That's not really what is going on in this problem -- you already have three specific distributions in mind and you want to give them weights. $\endgroup$ – mpr Oct 6 '13 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.