4
$\begingroup$

When characterizing an information measure one desires to have the following 'Grouping' property (cf., Cover&Thomas, Ch.2 exercise 46)

$$H(p_1, p_2,\dots, p_n)=H(p_1+p_2, p_3,\dots, p_n)+(p_1+p_2)H(\frac{p_1}{p_1+p_2},\frac{p_2}{p_1+p_2})$$

(a.k.a. recursive). An analogous Grouping axiom is employed for Renyi entropy in Jizba, Arimitzu. Can anybody give an intuitive meaning of it and why it is desired?

Also in the axiomatic characterization to inference based on entropy measures one has a set of properties(or axioms)(cf.,Shore and Johnson). Does the above Grouping property have any connection with any of the axioms of entropy based inference?

In general, is there any connection between the axioms in the axiomatic characterization of entropy(information) measures and the axioms in the axiomatic characterization of inference based on entropy measures? I know some of the connections say the symmetry property of entropy is related to the invariance property of inference, additivity of entropy is related to system independence of inference etc.

$\endgroup$

1 Answer 1

4
$\begingroup$

There is a simple interpretation of the above grouping property. Suppose your alphabet is $A, B, C,...$ where the letters have frequency $p_1, p_2, p_3, ..$ Now let $S$ be a random sequence of large length in your alphabet. Introduce a modified alphabet in which the letters $A$ and $B$ are merged into a new letter, $\alpha$. Thus $\alpha$ has frequency $p_1 + p_2$. Now let $S'$ be a random sequence of large length in the modified alphabet. The grouping property stipulates that the entropy $H(S)$ be equal to the entropy $H(S')$ plus the conditional entropy of predicting whether $\alpha$ corresponded originally to $A$ or $B$ in the old alphabet.

$\endgroup$
3
  • $\begingroup$ Thanks. Does anybody have the answer for the next question "Does the Grouping property have any connection with any of the axioms of entropy based inference?" $\endgroup$
    – Ashok
    Jun 10, 2011 at 8:54
  • $\begingroup$ There may be multiple subtly different axiomatic justifications for entropy-based inference. It's not clear which one in particular you are referring to. $\endgroup$ Jun 10, 2011 at 16:46
  • $\begingroup$ Is there any axiom which is used to justify entropy based inference which has connection with this Grouping property? $\endgroup$
    – Ashok
    Jun 11, 2011 at 4:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.