I've got a problem analytically obtaining the mutual information of a random variable with itself.

I have a set X of 7 balls: 3 red 2 blue 2 green

Therefore p(x) {3/7,2/7,2/7} I know that the entropy of my random variable X is H(X) = 1.55

I know that the mutual information of a random variable with itself or self information is just equal to its entropy: I(X,X) = H(X)

I run into the following problem: If I throw my balls into a box and want to grab two balls at a time I can compute the probability of obtaining every pair:

enter image description here

I assume this corresponds to the joint probability or p(X,X), as you can notice, the marginals correspond to p(X).

When I compute I(X,X) using either:

enter image description here


H(X,Y) = H(X) + H(Y) - H(X,Y)

I DO NOT obtain I(X,X) = H(X)


1 Answer 1


Your table of "joint probabilities" is not what you assume it to be. Joint probability $\Pr(X, X)$ means probability that $X=x$ and that $X=x$. Obviously $X=x$ is the same as $X=x$, so your table should have entries only on the diagonal. To prove it to ourselves, let's simulate:

X <- sample(1:3, 1000, replace = TRUE, prob = c(1, 5, 2))

prop.table(table(X, X))

##    X
## X       1     2     3
##   1 0.137 0.000 0.000
##   2 0.000 0.595 0.000
##   3 0.000 0.000 0.268

From your comment it seems that you misunderstand the general ideas (or notation) used in probability theory. Then we are talking about $\Pr(X=x)$ we mean probability that random variable $X$ takes value $x$, where $x$ stands for some particular value.

To make this more concrete, take one coin out of your wallet. This coin is not random, it is actually far from being random as it is very fixed metal object. What we consider as random, is the outcome of experiment, where we will be tossing this coin. It is random because, before we toss it, the outcome of such experiment is uncertain and unpredictable for us. Let's call the experiment of tossing this coin once as $A$. It potentially can have two outcomes: heads or tails ($h_A, t_A$ for simplicity). Now, when we are talking about $\Pr(A = h_A)$ we mean probability that in our experiment we will observe heads. Now imagine that we have two coins, and we toss them simultaneously. $\Pr(A = h_A, B = h_B)$ is the probability of observing heads in experiment $A$ and heads in experiment $B$. We may consider different experiments, e.g. let's toss the first coin two times and call outcome of the first toss as $A_1$, and outcome of the second toss as $A_2$. Now $\Pr(A_1 = h_{A_1}, A_2 = t_{A_2})$ is the probability that in the first toss we observed heads and in the second toss we observed tails. Notice that we used a single coin, but we denoted outcomes of the two experiments as two distinct random variables.

What follows, $\Pr(X = x, X = x)$ is the probability that when in the next few minutes you will toss the coin, that you took out of your wallet, it will land heads and that it will land heads, so it's a tautology. It is impossibility for $X$ to take two different values at the same time.

See also the joint distribution of x with..itself thread that extends this case to probability density functions.

  • $\begingroup$ I don't agree with your interpretation of joint probability. P(X,X) tells the probability of X = x with x in {X}. So p(X,X) could be p(1,2) or any other off-diagonal combination. Therefore the table of probabilities is not restricted to the diagonal. In any case it should only include the lower triangular matrix. $\endgroup$
    – jregalad
    Commented Jun 22, 2017 at 16:09
  • 1
    $\begingroup$ @jregalad it's not interpretation, it's definition. If you don't agree, then what is the probability that if you toss a coin it will land heads and tails in a single flip? If you tell me that it could be anything other then zero, then you must be using some very strange notion of probability. $\endgroup$
    – Tim
    Commented Jun 22, 2017 at 17:16
  • 1
    $\begingroup$ jregalad, it is precisely because you don't agree with this definition that you are getting the wrong result. If you would like a formal treatment, recall that $X$ is a real-valued map defined on the population $\Omega$. "$(X,X)$" means to compose that map with the one from $\mathbb{R}\to\mathbb{R}^2$ given by $x\to (x,x)$. The result is bivariate: it's a map from $\Omega$ into $\mathbb{R}^2$. By construction, it cannot ever take values off the diagonal, and so there can only be zero probability of off-diagonal values. $\endgroup$
    – whuber
    Commented Jun 22, 2017 at 20:52
  • $\begingroup$ @Tim I completely see it now. So easy to have misconceptions in statistics. Thanks a lot!!!!! $\endgroup$
    – jregalad
    Commented Jun 23, 2017 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.