0
$\begingroup$

Trying to understand how to derive that the self mutual information I(X,X) is equal to the entropy H(X), as stated on the wiki page wiki mutual information, "therefore H(X) = I(X;X)". More than that, I want to understand the derivation, and the meaning of P(X,X) if it is involved.

Start form the wiki formula for mutual information, $$ I(X;Y) = \sum_{x\in X} \sum_{y \in Y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)} $$

Changing this, substitute y -> x, writing what should be the formula for I(X,X) $$ I(X;X) = \sum_{x1\in X} \sum_{x2 \in X} p(x_1,x_2) \log \frac{p(x_1,x_2)}{p(x_1)p(x_2)} $$ it involves p(X,X).

I do not know what $p(X,X)$ means, and think I am missing some rules for simplifying this. Also I have not seen it discussed in books.

If possible please give a derivation of I(X,X)=H(X), without skipping/assuming any rules that are needed in this case but not whith I(X,Y), Y!=X.

Self studying

EDIT: corrected following whuber's comment.

Writing it more carefully, "P(X,X)" becomes $P(X=x_1,X=x_2)$ and probably this must be zero unless $x_1 = x_2$. Ok, so all terms of the double sum disappear unless $x_1 = x_2$, and then the missing rule must be: $$ p(x,x) = p(x) $$ ????????Is this true????????

First getting rid of all terms in the sum $x_1 \ne x_2$:: $$ I(X;X) = \sum_{x1 \in X} p(x_1) \log \frac{p(x_1,x_1)}{p(x_1)p(x_1)} $$ Then apply "p(x,x) = p(x)" $$ = \sum_{x1 \in X} p(x_1) \log \frac{p(x_1)}{p(x_1)p(x_1)} $$ $$ = \sum_{x1 \in X} p(x_1) \log \frac{1}{p(x_1)} = H(X) $$ Very happy now but please tell if the "p(x,x)=p(x)" rule is true

SECOND EDIT: Following whuber's last comment below, the "p(x,x)=p(x)" rule is true, and the relation is derived.

$\endgroup$
  • 4
    $\begingroup$ Could you please explain your notation? What are $x,$ $p,$ $I,$ and $H$ intended to be? Although we might guess these are related to probability densities, mutual information, and entropy, the fact that the formula appears to make no sense means we shouldn't be guessing about what you're asking. What is your source for this material? $\endgroup$ – whuber Dec 26 '19 at 22:00
  • 2
    $\begingroup$ You should also clarify (for yourself and for us) what random variables are at play here. In particular, you use the variable $x$ in two or three different capacities on the left- and right-hand sides of the equation. I suspect that you actually mean to calculate the mutual information of the random variable $X$ with itself, $I(X;X)$. $\endgroup$ – tddevlin Dec 26 '19 at 22:30
  • $\begingroup$ Yes this is a question about (self) mutual information and entropy. p() denotes probability. I have clarified the question and its notations $\endgroup$ – stochasticthoughtprocess Dec 27 '19 at 16:42
  • 1
    $\begingroup$ Re the formula following "changing this:" it makes no sense because you now ambiguously use "$x$" to refer to two different summation indices. When you change "$Y$" to "$X$" on the left hand side, the only modification you need to make on the right hand side is to change "$Y$" to "$X.$" The variable "$y$" must not be altered. This should cause you to modify the question itself, because the offending "$p(x,x)$" will never appear. $\endgroup$ – whuber Dec 27 '19 at 16:43
  • $\begingroup$ Thank you. I edited the post, closer now, but unfortunately there is still a p(x,x). Please see the new version of the question! $\endgroup$ – stochasticthoughtprocess Dec 28 '19 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.