Property of entropy

Question

When characterizing an information measure one desires to have the following 'Grouping' property (cf., Cover&Thomas, Ch.2 exercise 46)

$$H(p_1, p_2,\dots, p_n)=H(p_1+p_2, p_3,\dots, p_n)+(p_1+p_2)H(\frac{p_1}{p_1+p_2},\frac{p_2}{p_1+p_2})$$

(a.k.a. recursive). An analogous Grouping axiom is employed for Renyi entropy in Jizba, Arimitzu. Can anybody give an intuitive meaning of it and why it is desired?

Also in the axiomatic characterization to inference based on entropy measures one has a set of properties(or axioms)(cf.,Shore and Johnson). Does the above Grouping property have any connection with any of the axioms of entropy based inference?

In general, is there any connection between the axioms in the axiomatic characterization of entropy(information) measures and the axioms in the axiomatic characterization of inference based on entropy measures? I know some of the connections say the symmetry property of entropy is related to the invariance property of inference, additivity of entropy is related to system independence of inference etc.

Answer 1

There is a simple interpretation of the above grouping property. Suppose your alphabet is $A, B, C,...$ where the letters have frequency $p_1, p_2, p_3, ..$ Now let $S$ be a random sequence of large length in your alphabet. Introduce a modified alphabet in which the letters $A$ and $B$ are merged into a new letter, $\alpha$. Thus $\alpha$ has frequency $p_1 + p_2$. Now let $S'$ be a random sequence of large length in the modified alphabet. The grouping property stipulates that the entropy $H(S)$ be equal to the entropy $H(S')$ plus the conditional entropy of predicting whether $\alpha$ corresponded originally to $A$ or $B$ in the old alphabet.