4
$\begingroup$

How can be proven that for random variables $A$ and $B$, and $C = A + B$,

$$H(C\mid A) = H(B\mid A).$$

Also, would it be possible to determine if $H(C)$ would be greater than $H(A)$?

$\endgroup$
4
  • $\begingroup$ any ideas will be appreciated. $\endgroup$
    – whynot
    Commented Oct 26, 2012 at 1:22
  • $\begingroup$ It is a homework, isn't it? Just think about what happens when $A=a$. What is the difference between $C|A=a$ and $B|A=a$. Anyway, looking at en.wikipedia.org/wiki/Convolution may help (if two probabilities have densities, the density of their sum is just a convolution of their densities). $\endgroup$ Commented Nov 3, 2012 at 0:15
  • $\begingroup$ Differential or Shannon entropy? The sum of absolutely continuous variables need not even be absolutely continuous. Given the poor behavior of differential entropy under sums, I would doubt there is such a relationship for differential entropy. $\endgroup$
    – cantorhead
    Commented Feb 3, 2018 at 19:21
  • $\begingroup$ Some observations. The problem seems equivalent to showing $H(A,A+B)=H(A,B)$. Also, is there anything special about addition here? Would the relationship $H(A,AB)=H(A,B)$ also hold under suitable conditions? $\endgroup$
    – cantorhead
    Commented Feb 3, 2018 at 19:24

1 Answer 1

0
$\begingroup$

This is an elementary exercise, when the random variables take only a numerable number of values: $A(\omega)\in \mathcal{A}$, $B(\omega)\in \mathcal{B}$ and $C(\omega)\in \mathcal{C}$. In that case:

$$H(C|A) = -\sum_{a\in\mathcal{A}}\sum_{c\in\mathcal{C}} p_A(a)p_{C|A}(c|a)\ln p_{C|A}(c|a), \tag{*}\label 1$$

Since $C = A + B$, we have that:

$$p_{C|A}(c|a) = p_{B|A}(c-a|a)$$

And substituting this into $\eqref 1$, the conditional entropy is:

$$H(C|A) = -\sum_{a\in\mathcal{A}}\sum_{c\in\mathcal{C}} p_A(a)p_{B|A}(c-a|a)\ln p_{B|A}(c-a|a), \tag{**}\label 2$$

restructuring the second sum over values of $b = c-a$, we have:

$$H(C|A) = -\sum_{a\in\mathcal{A}} p_A(a) \sum_{b\in\mathcal{B}} p_{B|A}(b|a)\ln p_{B|A}(b|a) = -\sum_{a\in\mathcal{A}} p_A(a) H(B|a) = H(B|A).$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.