Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to:

$$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$

with only the natural constraint of $p_1+p_2+p_3=1$

Found two compelling arguments using entropy, which paraphrasing for this problem:


We would like to maximize the Shannon Entropy

$$H_S(P)= - \sum \ p_i \log(p_i)) $$

subject to the natural constraint and the $p$'s bounded by the inequalities. As $p^*=1/3$ for all probabilities is both the global optimimum of the entropy function with the natural constraint and satisfies all inequalities we would declare the answer

$p=(1/3, 1/3, 1/3)$



Jayne's Principle of Maximum Entropy is only valid with linear constraints, the use of inequalities is not directly applicable to Shannon entropy. We would get the same answer as above for any set of inequalities as long as $p^*=1/3$ is contained within each inequality. The fact that $0 \le p_3 \le .4$ or $0.33331 \le p_3 \le 0.33334$ would be immaterial to the above although the last one is most informative. Subject to only the natural constraint the principle of indifference is not on the probabilities themselves, but on where they are in the inequality. The inequality $0.33331 \le p_3 \le 0.33334 $ gives a lot more information than $ 0.31 \le p_3 \le .34 $ than $0 \le p_3 \le 0.4$. We must build up a measure of uncertainty from first principles that implicitly takes those inequalities, and information they are stating, into account. This is a special case of the generalized maximum entropy principle with inequalities on each probability only

$$a_i \le p_i \le b_i $$

We should maximize

$$H_K(P)= \left( - \sum \ (p_i-a_i) \log(p_i-a_i)) \right) + \left( - \sum \ (b_i-p_i) \log(b_i-p_i)) \right) $$

subject to the constraints. If the normalization constraint is the only constraint the optimization reduces to the fact that $(p_i-a_i)/(b_i-a_i)$ should be the same for all probabilities within their respective inequalities. We are maximimally uncertain of where in the inequality they should be, and by an extension of Laplace Principle of Insufficient Reason we should have them all in the same proportion within their intervals.

For the problem above we would have $(p_i-a_i)/(b_i-a_i)=0.5$ yielding

$p=(0.25, 0.4, 0.35)$

Each probability is in the same proportion within their interval, in this case halfway.

In most optimization books and papers I've seen when discussing maximizing the entropy it is treated as any other convex optimization

: \begin{align} &\underset{x}{\operatorname{maximize}}& & f(x) \\ &\operatorname{subject\;to} & &lb_i \le x_i \le ub_i, \quad i = 1,\dots,m \\ &&&h_i(x) = 0 , \quad i = 1, \dots,p. \end{align}

with $f(x)$ as Shannon Entropy and the inequalities box in the search space.

Kapur seems to argue that the bounds of the inequalities themselves provide information and should be taken into account, with a new optimization function subject to linear constraints

: \begin{align} &\underset{x}{\operatorname{maximize}}& & g(x) \\ &\operatorname{subject\;to} &&h_i(x) = 0 , \quad i = 1, \dots,p. \end{align}

Although we only used the natural constraint, both optimizations can be expressed in terms of Lagrange Multipliers for more additional constraints and more probabilities.

The question I have is when is either argument applicable? I can understand Jaynes argument, but it does seem to ignore the boundedness of the inequalities as long as the global minimum is contained within them. (If not contained the optimization would have some on the boundary of the inequality). Kapur also makes sense, the probabilities should be maximally uncertain where in the inequality they are, subject to the equality constraints.

Additionally, wouldn't all probabilities have the bounds $0 \le p_i \le 1$? Or is the upper limit implicit in the normalization constraint and $p_i \ge 0$ inequality which is usually seen in Maximum Entropy problems. If $a_i=0$ and $b_i$ unspecified, it seems $H_K$ reduces to $H_S$

  • $\begingroup$ What would Kapur do in a case of more complex constraints, such as $\sum_{i=1}^3 (p_i-0.4)^2\le 0.3^2$? Has he offered a solution in cases like that? I have difficulty making sense of his concepts of having "more" information about some of the $p_i$ compared to others. As far as the quotation is concerned, the argument founders at the point where it becomes vague and appeals to an unsubstantiated, meaningless idea "an extension of Laplace Principle of Insufficient Reason." Does Kapur ever articulate this "extension"? $\endgroup$ – whuber Jun 5 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.