3
$\begingroup$

This is something from Chris Bishop's book and I want to confirm that I have understood this correctly.

So, suppose we have a joint distribution between two RVs as: $$ P(a, b) = P(a|b)P(b) = P(b|a)P(a) $$ Now, suppose that $b$ is observed, can I write the joint distribution as:

$$ P(a, b) = P(a|b)P(b) = P(a|b) $$ Since, $b$ is observed, $P(b)$ is 1.

$\endgroup$
  • $\begingroup$ There is no conditional independence here. $\endgroup$ – Dilip Sarwate May 7 '17 at 13:39
1
$\begingroup$

This question is complicated by the choice of notation in the book you are using. With a little more formalism in the notation, the matters that puzzle you are resolved easily.

Consider discrete random variables $X$ and $Y$ with joint distribution $$ p_{X,Y}(0,0) = P\{X=0, Y=0\} = 0.1, ~~p_{X,Y}(1,0) =P\{X=1, Y=0\} = 0.2,\\ p_{X,Y}(0,1) = P\{X=0, Y=1\} = 0.3, ~~p_{X,Y}(1,1) = P\{X=1, Y=1\} = 0.4. $$ Then, $X$ is a Bernoulli random variable with parameter $0.6$, that is, $$p_X(1) = P\{X=1\} = 0.6, ~~p_X(0) = P\{X=0\} = 0.4,$$ while $Y$ is a Bernoulli random variable with parameter $0.7$, that is, $$p_Y(1) = P\{Y=1\} = 0.7, ~~p_Y(0) = P\{Y=0\} = 0.3.$$ Now, given that $Y=1$, the conditional distribution of $X$ is $$p_{X\mid Y=1}(1\mid Y=1) = P\{X = 1\mid Y = 1\} = \frac{P\{X=1,Y=1\}}{P\{Y=1\}} = \frac{0.4}{0.7} = \frac 47,\\ p_{X\mid Y=1}(0\mid Y=1) = P\{X = 0\mid Y = 1\} = \frac{P\{X=0,Y=1\}}{P\{Y=1\}} = \frac{0.3}{0.7} = \frac 37,$$ that is, conditioned on the event $Y=1$, the conditional distribution of $X$ is Bernoulli with parameter $\frac 47$. Similarly, conditioned on the event $Y=0$, the conditional distribution of $X$ is Bernoulli with parameter $\frac 23$. Work out the details of this last calculation to make sure that you understand. Then, work out all the details of the calculation that conditioned on $X=1$, the conditional distribution of $Y$ is Bernoulli with parameter $\frac 23$ while conditioned on $X=0$, the conditional distribution of $Y$ is Bernoulli with parameter $\frac 34$.

Next, verify for yourself that for all choices of $i, j \in \{0,1\}$, it is true that $$p_{X,Y}(i,j) = p_{X\mid Y = j}(i\mid Y=j)p_Y(j) = p_{Y\mid X=i}(j\mid X=i)p_X(i).$$ There is no such thing as $p(a,b) = p(a\mid b)p(b) = p(a\mid b)$ as you think "because $p(b) = 1$ since $b$ has occurred". $p(b)$ does not change its value "when $b$ has occurred" because random variables always occur on each and every trial of the experiment; what changes is the value taken on by the random variable on each trial.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

No, since b is observed doesn't mean that P(b)=1. If you were to repeat the experiment then it wouldn't mean that b will be observed again, because that's what P(b)=1 means. The fact that you observed doesn't make it certain.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Let's take $A$ and $B$ to be binary variables to make things simple: $A$ must be equal to either $a_1$ or $a_2$, and $B$ must be equal to either $b_1$ or $b_2$. Suppose that our information state $I$ prior to learning the value of $B$ gives the following joint PDF between $A$ and $B$:

\begin{align*} P(A=a_1,B=b_1\;|\;I\,) & = p_{11} \\ P(A=a_1,B=b_2\;|\;I\,) & = p_{12} \\ P(A=a_2,B=b_1\;|\;I\,) & = p_{21} \\ P(A=a_2,B=b_2\;|\;I\,) & = p_{22} \end{align*}

Now suppose we learn that $B = b_1$. Then we can apply Bayes' Theorem to give the posterior PDF over $A$:

$$ P(A = a_i\;|\;B = b_1,I\,) = \frac{P(A = a_i,B = b_1\;|\;I\,)}{ P(B = b_1\;|\;I\,) } $$

Note that $P(B = b_1\;|\;I\,)$ is conditioned on our initial information $I$, so it does not necessarily equal $1$. Now we can introduce the probabilities of the joint PDF:

$$ P(A = a_i\;|\;B = b_1,I\,) = \frac{p_{i1}}{p_{11} + p_{21}} $$

In our posterior information state $\{B = b_1, I\}$, we have $P(B = b_1\;|\;B = b_1, I\,) = 1$, which is quite persuasively evident when one writes it like that! Happily, the posterior probabilities of the states of $A$ sum to $1$:

$$ \sum_{i = 1}^{2} \frac{p_{i1}}{p_{11} + p_{21}} = \frac{p_{11} + p_{21}}{p_{11} + p_{21}} = 1 $$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.