For conditional probability $P(D_1, D_2|W)$, suppose $D_1$ and $D_2$ are dependent (I mean not i.i.d.), then I am a bit confused whether $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W)$$ or $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W, D_1)$$

From Bayes theorem, the latter one is correct, but from logical thinking, I cannot tell what is wrong for the first formula, since its logical meaning $D_1$ happens under conditional of $W$, at the same time $D_2$ happens under conditional of $W$, seems correct to represent the logical meaning of $P(D_1, D_2|W)$, which is $D_1$ and $D_2$ happens at the same time under the condition of W?

What is wrong in my above analysis, which result in wrong decision $$P(D_1, D_2|W) = P(D_1|W) \ P (D_2|W)$$ when $D_1$ and $D_2$ are not independent?


Basically, you are referring to conditional independence. Imagine that we have three events, $A,B,C$, we say that $A$ and $B$ are conditionally independent given $C$ if

$$ \Pr(A \cap B \mid C) = \Pr(A \mid C) \, \Pr(B \mid C) $$

so by using the first formula you are assuming conditional independence, what may, or may not be true for your data. This is related to the idea of exchangeability, in Bayesian statistics we often assume that our data is independent and identically distributed conditionally on parameters (see also the O'Neill, 2009 paper that Wikipedia refferes to).

To learn more refer to Wikipedia article and math.stackexchange.com thread that go into more details and provide multiple worked examples.

O'Neill, B. (2009). Exchangeability, Correlation and Bayes' Effect. International Statistical Review 77(2), 241–250.

  • $\begingroup$ Thanks Tim, I think if I can write $Pr(A,B|C) = Pr(A|C) * Pr (B|C)$, it means A and B are independent, correct? So, I think saying A and B are independent (under any condition) is ok, why you say that A and B are conditionally independent? $\endgroup$ – Lin Ma Sep 16 '16 at 6:40
  • 1
    $\begingroup$ @LinMa no. It means that they are independent conditionally on $C$. Unconditionally they can be dependent. Check e.g. books.google.pl/… and the links I provided. $\endgroup$ – Tim Sep 16 '16 at 7:15
  • 1
    $\begingroup$ @LinDa have you checked the links I posted in my answer and comment? They contain multiple examples. Also if you are not familiar with basic probability topics then maybe you should start with portability handbook? $\endgroup$ – Tim Sep 17 '16 at 11:30
  • 1
    $\begingroup$ @LinMa please check also the links in my answer. Together with the link in the comment they extend and illustrate what I've written in my answer. They also give examples for what you're asking. $\endgroup$ – Tim Sep 18 '16 at 6:54
  • 1
    $\begingroup$ @LinMa it is hard to give clear and intuitive example of conditional independence. You could check this one stats.stackexchange.com/questions/125683/… it is not about conditional independence, but shows how conditioning changes relationships. I'm sorry but it's hard for me to give better explanations then the ones given. $\endgroup$ – Tim Sep 22 '16 at 19:45

It may be helpful to break the conditional probability in terms of the joint and marginal probabilities.

Starting from the definition of conditional probability, we have \begin{align} p(x,y \mid z) &= \frac{p(x,y,z)}{p(z)} \\ &= \frac{p(x,y,z)}{p(y,z)} \, \frac{p(y,z)}{p(z)} \\ &= p(x \mid y,z) \, p(y \mid z) \end{align} which shows that the correct formula is your second one. This is sometimes called the chain rule of probability. Note that it does not require any independence.

(Note that this is not Bayes' theorem, which uses the chain rule, but is fundamentally about "swapping variables across the conditional", i.e. $p(z|x,y)$ would be on the right hand side.)

As Tim notes, your alternate expression only holds in the case that $x$ and $y$ are conditionally independent, given $z$. Two variables $x$ and $y$ are independent if knowing the value of one gives no information about the value of the other, i.e. $p(x|y)=p(x)$ (and so $p(y|x)=p(y)$, by Bayes' theorem).

Now it could be that knowing $y$ does give information about $x$, but if $z$ also gives that same information, then given $z$ the value of $y$ gives no new information about $x$. This conditional independence would be expressed mathematically as $$p(x|z,y)=p(x|z)$$ This gives your alternate expression, when substituted into the more general formula above.

  • $\begingroup$ Excellent explanation GeoMatt22, studifed the "chain rule of probability" you referred and it seems a chain rule for joint probability (represent joint probability by multiply conditional probability). But my question is about how to represent conditional probability by other conditional probabilities, not sure if chain rule applies here? Thanks. $\endgroup$ – Lin Ma Sep 16 '16 at 6:37
  • 1
    $\begingroup$ @LinMa your question asks how to factor a conditional joint probability into a product of conditional probabilities. The chain rule tells you the answer, as I wrote above. The chain rule always applies. Conditional independence only applies sometimes. For more on conditional independence, you can see this SE post (referenced in the Wikipedia link I gave). Note "joint" just means there is more than 1 variable left of the "|", and "conditional" means there are more than 0 variables to its right. $\endgroup$ – GeoMatt22 Sep 20 '16 at 0:51
  • $\begingroup$ Thanks GeoMatt, for SE post you ment post by @joriki? $\endgroup$ – Lin Ma Sep 20 '16 at 6:57
  • 1
    $\begingroup$ @LinMa my comment had a link in it, with the text "this". It is to a related question on the Math SE. $\endgroup$ – GeoMatt22 Sep 20 '16 at 12:52
  • $\begingroup$ Thanks GeoMatt, I like the post you referred and vote up. But I disagree with the example of "a blue die and a red die", under the condition of knowing their sum is even, and if we know value of one die, we know there are only 3 (other than 6) possible values of the other die, it is 100% correct, but how does it links to independence of two die? I think it (know sum is even) does not break the assumption of two die are independent (for independent I mean the two die makes choices of their values independently)? Maybe we have different definition of what means independent here? $\endgroup$ – Lin Ma Sep 21 '16 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.