Maximum Entropy with bounded constraints

Question

Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to:

$$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$

with only the natural constraint of $p_1+p_2+p_3=1$

Found two compelling arguments using entropy, which paraphrasing for this problem:

Jaynes

We would like to maximize the Shannon Entropy

$$H_S(P)= - \sum \ p_i \log(p_i)) $$

subject to the natural constraint and the $p$'s bounded by the inequalities. As $p^*=1/3$ for all probabilities is both the global optimimum of the entropy function with the natural constraint and satisfies all inequalities we would declare the answer

$p=(1/3, 1/3, 1/3)$

.

Kapur

Jayne's Principle of Maximum Entropy is only valid with linear constraints, the use of inequalities is not directly applicable to Shannon entropy. We would get the same answer as above for any set of inequalities as long as $p^*=1/3$ is contained within each inequality. The fact that $0 \le p_3 \le .4$ or $0.33331 \le p_3 \le 0.33334$ would be immaterial to the above although the last one is most informative. Subject to only the natural constraint the principle of indifference is not on the probabilities themselves, but on where they are in the inequality. The inequality $0.33331 \le p_3 \le 0.33334 $ gives a lot more information than $ 0.31 \le p_3 \le .34 $ than $0 \le p_3 \le 0.4$. We must build up a measure of uncertainty from first principles that implicitly takes those inequalities, and information they are stating, into account. This is a special case of the generalized maximum entropy principle with inequalities on each probability only

$$a_i \le p_i \le b_i $$

We should maximize

$$H_K(P)= \left( - \sum \ (p_i-a_i) \log(p_i-a_i)) \right) + \left( - \sum \ (b_i-p_i) \log(b_i-p_i)) \right) $$

subject to the constraints. If the normalization constraint is the only constraint the optimization reduces to the fact that $(p_i-a_i)/(b_i-a_i)$ should be the same for all probabilities within their respective inequalities. We are maximimally uncertain of where in the inequality they should be, and by an extension of Laplace Principle of Insufficient Reason we should have them all in the same proportion within their intervals.

For the problem above we would have $(p_i-a_i)/(b_i-a_i)=0.5$ yielding

$p=(0.25, 0.4, 0.35)$

Each probability is in the same proportion within their interval, in this case halfway.

In most optimization books and papers I've seen when discussing maximizing the entropy it is treated as any other convex optimization

: \begin{align} &\underset{x}{\operatorname{maximize}}& & f(x) \\ &\operatorname{subject\;to} & &lb_i \le x_i \le ub_i, \quad i = 1,\dots,m \\ &&&h_i(x) = 0 , \quad i = 1, \dots,p. \end{align}

with $f(x)$ as Shannon Entropy and the inequalities box in the search space.

Kapur seems to argue that the bounds of the inequalities themselves provide information and should be taken into account, with a new optimization function subject to linear constraints

: \begin{align} &\underset{x}{\operatorname{maximize}}& & g(x) \\ &\operatorname{subject\;to} &&h_i(x) = 0 , \quad i = 1, \dots,p. \end{align}

Although we only used the natural constraint, both optimizations can be expressed in terms of Lagrange Multipliers for more additional constraints and more probabilities.

The question I have is when is either argument applicable? I can understand Jaynes argument, but it does seem to ignore the boundedness of the inequalities as long as the global minimum is contained within them. (If not contained the optimization would have some on the boundary of the inequality). Kapur also makes sense, the probabilities should be maximally uncertain where in the inequality they are, subject to the equality constraints.

Additionally, wouldn't all probabilities have the bounds $0 \le p_i \le 1$? Or is the upper limit implicit in the normalization constraint and $p_i \ge 0$ inequality which is usually seen in Maximum Entropy problems. If $a_i=0$ and $b_i$ unspecified, it seems $H_K$ reduces to $H_S$

Answer 1

From what I understood, it seems to me that Kapur and Jaynes don't understand the inegality constraints as the same piece of information.

I tried to think of a real life example to illustrate that difference :

Imagine a bakery only selling almond-croissant and chocolate-donuts and which had 100 customers on a given day. You are intersted in the proportion $p$ of customers that bought a chocolate-donut rather than the almond-croissant.

With no information, maximum entropy would make you give $p = 0.5$ as the least informative guess.

Now, if you learn that the bakery only had 80 donuts and 90 croissant available that day, you get the constraint: $0.1 \leq p \leq 0.8$ (supposing that customers were hungry so that they bought the other option if theirs was sold out). Should you change your guess ? I think not, since the constraint is independant of the custumer's choice. So you would go with Jaynes' approach (maximizing usual entropy under the contraint) and still give $p = 0.5$ as the least informative guess.

But if instead of the last information, you were told that $20$ customers were allergic to chocolate so couldn't take a donut and 10 were allergic to almonds so couldn't take a croissant. You would also get the constraint that $0.1 \leq p \leq 0.8$. Should you change your guess ? Yes, a more appropriate guess would be $p = 0.45$, which corresponds to supposing that half the customers who had no allergies (and thus really had a choice) chose a croissant rather than the chocolate-donut. So you would go with Kapur's approach.

With only one probability $p$, Kapur's approach is equivalent to applying Jaynes' approach to the transform probability $\frac{p - a}{b - a}$ and this transform proportion is meaningful: in the example above where $p$ denotes the proportion of customers who chose a donut, $\frac{p - a}{b - a}$ denotes the proportion of non allergic customers who chose a donut. For several probabilities, I am not sure if this is equivalent, but it does go into this direction. In particular Kapur's modified entropy has the nice property that each term of the sum is invariant if you change $p_i$ into $b_i - p_i + a_i$.

I hope this helped a bit.