# Conditional independence and joint distribution

This is something from Chris Bishop's book and I want to confirm that I have understood this correctly.

So, suppose we have a joint distribution between two RVs as: $$P(a, b) = P(a|b)P(b) = P(b|a)P(a)$$ Now, suppose that $b$ is observed, can I write the joint distribution as:

$$P(a, b) = P(a|b)P(b) = P(a|b)$$ Since, $b$ is observed, $P(b)$ is 1.

• There is no conditional independence here. – Dilip Sarwate May 7 '17 at 13:39

This question is complicated by the choice of notation in the book you are using. With a little more formalism in the notation, the matters that puzzle you are resolved easily.

Consider discrete random variables $X$ and $Y$ with joint distribution $$p_{X,Y}(0,0) = P\{X=0, Y=0\} = 0.1, ~~p_{X,Y}(1,0) =P\{X=1, Y=0\} = 0.2,\\ p_{X,Y}(0,1) = P\{X=0, Y=1\} = 0.3, ~~p_{X,Y}(1,1) = P\{X=1, Y=1\} = 0.4.$$ Then, $X$ is a Bernoulli random variable with parameter $0.6$, that is, $$p_X(1) = P\{X=1\} = 0.6, ~~p_X(0) = P\{X=0\} = 0.4,$$ while $Y$ is a Bernoulli random variable with parameter $0.7$, that is, $$p_Y(1) = P\{Y=1\} = 0.7, ~~p_Y(0) = P\{Y=0\} = 0.3.$$ Now, given that $Y=1$, the conditional distribution of $X$ is $$p_{X\mid Y=1}(1\mid Y=1) = P\{X = 1\mid Y = 1\} = \frac{P\{X=1,Y=1\}}{P\{Y=1\}} = \frac{0.4}{0.7} = \frac 47,\\ p_{X\mid Y=1}(0\mid Y=1) = P\{X = 0\mid Y = 1\} = \frac{P\{X=0,Y=1\}}{P\{Y=1\}} = \frac{0.3}{0.7} = \frac 37,$$ that is, conditioned on the event $Y=1$, the conditional distribution of $X$ is Bernoulli with parameter $\frac 47$. Similarly, conditioned on the event $Y=0$, the conditional distribution of $X$ is Bernoulli with parameter $\frac 23$. Work out the details of this last calculation to make sure that you understand. Then, work out all the details of the calculation that conditioned on $X=1$, the conditional distribution of $Y$ is Bernoulli with parameter $\frac 23$ while conditioned on $X=0$, the conditional distribution of $Y$ is Bernoulli with parameter $\frac 34$.

Next, verify for yourself that for all choices of $i, j \in \{0,1\}$, it is true that $$p_{X,Y}(i,j) = p_{X\mid Y = j}(i\mid Y=j)p_Y(j) = p_{Y\mid X=i}(j\mid X=i)p_X(i).$$ There is no such thing as $p(a,b) = p(a\mid b)p(b) = p(a\mid b)$ as you think "because $p(b) = 1$ since $b$ has occurred". $p(b)$ does not change its value "when $b$ has occurred" because random variables always occur on each and every trial of the experiment; what changes is the value taken on by the random variable on each trial.

No, since b is observed doesn't mean that P(b)=1. If you were to repeat the experiment then it wouldn't mean that b will be observed again, because that's what P(b)=1 means. The fact that you observed doesn't make it certain.

Let's take $A$ and $B$ to be binary variables to make things simple: $A$ must be equal to either $a_1$ or $a_2$, and $B$ must be equal to either $b_1$ or $b_2$. Suppose that our information state $I$ prior to learning the value of $B$ gives the following joint PDF between $A$ and $B$:

\begin{align*} P(A=a_1,B=b_1\;|\;I\,) & = p_{11} \\ P(A=a_1,B=b_2\;|\;I\,) & = p_{12} \\ P(A=a_2,B=b_1\;|\;I\,) & = p_{21} \\ P(A=a_2,B=b_2\;|\;I\,) & = p_{22} \end{align*}

Now suppose we learn that $B = b_1$. Then we can apply Bayes' Theorem to give the posterior PDF over $A$:

$$P(A = a_i\;|\;B = b_1,I\,) = \frac{P(A = a_i,B = b_1\;|\;I\,)}{ P(B = b_1\;|\;I\,) }$$

Note that $P(B = b_1\;|\;I\,)$ is conditioned on our initial information $I$, so it does not necessarily equal $1$. Now we can introduce the probabilities of the joint PDF:

$$P(A = a_i\;|\;B = b_1,I\,) = \frac{p_{i1}}{p_{11} + p_{21}}$$

In our posterior information state $\{B = b_1, I\}$, we have $P(B = b_1\;|\;B = b_1, I\,) = 1$, which is quite persuasively evident when one writes it like that! Happily, the posterior probabilities of the states of $A$ sum to $1$:

$$\sum_{i = 1}^{2} \frac{p_{i1}}{p_{11} + p_{21}} = \frac{p_{11} + p_{21}}{p_{11} + p_{21}} = 1$$