Shannon entropy with regards to independent random variables

Question

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific problem I'm referring to:

$\space$

Consider three independent random variables $u$, $v$, and $w$ with entropies $H_u$, $H_v$, $H_w$. Let,

$$X \equiv (U,\ V)$$ $$Y \equiv (V,\ W)$$

What is $H(X, Y)$, $H(X | Y)$, and $I(X; Y)$?

$\space$

Here's what I've come up with so far:

Since the random variables $u$, $v$, and $w$ are independent,

$$P(X) = P(U)P(V)$$ $$P(Y) = P(V)P(W)$$

And since $$P(X, Y) = P(X|Y)P(Y)$$ $$P(X|Y) = \frac{P(Y|X)P(U)}{P(W)}$$

But I'm not sure how to progress further from here... The solution my instructor provided for this particular problem in the textbook I'm using (Information Theory, Inference, and Learning Algorithms) is that:

$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$

$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$

And with this result the solution is fairly easy to derive.

$\space$

What I'm wondering is, where did the $x_2 = y_1$ come from, and how were the results for those probability distributions come to be? The approach I was taking was causing me to go in circles without any real results.

Thank you.

Answer 1

Since $X = (U, V)$ and $Y = (V, W)$ share the variable $V$ in any observation of both $X = (x_1,x_2) = (u, v)$ and $Y= (y_1,y_2) = (v, w),$ there is only one fixed value for $V=v$ such that $x_2 = v = y_1$.

Thus, $P\big((X,Y) = (x_1,x_2,y_1,y_2)\big)$ has value $0$ whenever $x_2 \neq y_1,$ and when $x_2$ and $y_1$ are equal (and have common value $v$) $P\big((X,Y) = (x_1,v,v,y_2)\big) = P(U=x_1)P(V=v)P(W=y_2)$