conditional probability of dependent events

Question

For conditional probability $P(D_1, D_2|W)$, suppose $D_1$ and $D_2$ are dependent (I mean not i.i.d.), then I am a bit confused whether $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W)$$ or $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W, D_1)$$

From Bayes theorem, the latter one is correct, but from logical thinking, I cannot tell what is wrong for the first formula, since its logical meaning $D_1$ happens under conditional of $W$, at the same time $D_2$ happens under conditional of $W$, seems correct to represent the logical meaning of $P(D_1, D_2|W)$, which is $D_1$ and $D_2$ happens at the same time under the condition of W?

What is wrong in my above analysis, which result in wrong decision $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W)$$ when $D_1$ and $D_2$ are not independent?

Answer 1

Basically, you are referring to conditional independence. Imagine that we have three events, $A,B,C$, we say that $A$ and $B$ are conditionally independent given $C$ if

$$ \Pr(A \cap B \mid C) = \Pr(A \mid C) \, \Pr(B \mid C) $$

so by using the first formula you are assuming conditional independence, what may, or may not be true for your data. This is related to the idea of exchangeability, in Bayesian statistics we often assume that our data is independent and identically distributed conditionally on parameters (see also the O'Neill, 2009 paper that Wikipedia refferes to).

To learn more refer to Wikipedia article and math.stackexchange.com thread that go into more details and provide multiple worked examples.

---

O'Neill, B. (2009). Exchangeability, Correlation and Bayes' Effect. International Statistical Review 77(2), 241–250.

Answer 2

It may be helpful to break the conditional probability in terms of the joint and marginal probabilities.

Starting from the definition of conditional probability, we have \begin{align} p(x,y \mid z) &= \frac{p(x,y,z)}{p(z)} \\ &= \frac{p(x,y,z)}{p(y,z)} \, \frac{p(y,z)}{p(z)} \\ &= p(x \mid y,z) \, p(y \mid z) \end{align} which shows that the correct formula is your second one. This is sometimes called the chain rule of probability. Note that it does not require any independence.

(Note that this is not Bayes' theorem, which uses the chain rule, but is fundamentally about "swapping variables across the conditional", i.e. $p(z|x,y)$ would be on the right hand side.)

As Tim notes, your alternate expression only holds in the case that $x$ and $y$ are conditionally independent, given $z$. Two variables $x$ and $y$ are independent if knowing the value of one gives no information about the value of the other, i.e. $p(x|y)=p(x)$ (and so $p(y|x)=p(y)$, by Bayes' theorem).

Now it could be that knowing $y$ does give information about $x$, but if $z$ also gives that same information, then given $z$ the value of $y$ gives no new information about $x$. This conditional independence would be expressed mathematically as $$p(x|z,y)=p(x|z)$$ This gives your alternate expression, when substituted into the more general formula above.