I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific problem I'm referring to:


Consider three independent random variables $u$, $v$, and $w$ with entropies $H_u$, $H_v$, $H_w$. Let,

$$X \equiv (U,\ V)$$ $$Y \equiv (V,\ W)$$

What is $H(X, Y)$, $H(X | Y)$, and $I(X; Y)$?


Here's what I've come up with so far:

Since the random variables $u$, $v$, and $w$ are independent,

$$P(X) = P(U)P(V)$$ $$P(Y) = P(V)P(W)$$

And since $$P(X, Y) = P(X|Y)P(Y)$$ $$P(X|Y) = \frac{P(Y|X)P(U)}{P(W)}$$

But I'm not sure how to progress further from here... The solution my instructor provided for this particular problem in the textbook I'm using (Information Theory, Inference, and Learning Algorithms) is that:

$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$

$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$

And with this result the solution is fairly easy to derive.


What I'm wondering is, where did the $x_2 = y_1$ come from, and how were the results for those probability distributions come to be? The approach I was taking was causing me to go in circles without any real results.

Thank you.

  • $\begingroup$ Does it help to clarify if you try to express $P(X=(x_1, x_2), Y=(y_1, y_2))$ in terms of $P(U=something), P(V=something), P(W=something)$ ? $\endgroup$ – Juho Kokkala Jun 15 '18 at 6:56
  • $\begingroup$ Yes, that would be the typical scenario that I would imagine. However, the textbook I'm using doesn't say anything about that, nor does it mention anything about there being two outcomes. I don't know where my instructor got the idea from either... $\endgroup$ – Seankala Jun 15 '18 at 7:49
  • $\begingroup$ What do you mean by two outcomes? It says $X=(U,V)$, so $X$ clearly consists of two "components", whatever they are (e.g., if $U=foo$ and $V=5$, then $X=(foo, 5)$, in other words, $x_1=foo$ and $x_2=5$). $\endgroup$ – Juho Kokkala Jun 15 '18 at 8:01

Since $X = (U, V)$ and $Y = (V, W)$ share the variable $V$ in any observation of both $X =x$ and $Y=y$ there is only one fixed value for $V=v$ so that that $x_2 = v = y_1$.

Thus, any outcome where $x_2 \neq y_1$ is impossible and gets zero probability.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.