If two or more variables A, B, C, etc. are jointly mutually independent of one another, does this imply that that they are also conditionally independent given some set of conditioning variables X, Y, Z, etc.?

If you test variables A, B, C,, etc., and the hypothesis of independence is not rejected, are you generally safe in assuming that these variables will also survive a test of conditional independence given X, Y, Z, etc.? Is it safe enough that you do not need to bother running the latter test? Why or why not?



Consider three boolean variables: A, B, X where X and A are i.i.d. Bernoulli with probabilty 0.5, while B = X $\oplus$ A (that is, B is equal to the xor of X and A).

It's easy to show that B is also Bernoulli distributed with probabilty 0.5, and A and B are mutually independent, though obviously they aren't conditionally independent given X.

  • $\begingroup$ Thanks Antonio! So, it is funny. In the context of log-linear models with a high number of dimensions, people talk about building the model up from the bottom or down from the top. But given this result, it seems like the "up from the bottom" approach is invalid, since you do not really know how strong the relationship is between any two variables until you have a complete model. $\endgroup$ – andrewH Mar 5 '13 at 2:40
  • $\begingroup$ It depends. Naive Bayes models make conditional independence assumption between variables, while models trained with logistic regression don't. $\endgroup$ – Antonio Valerio Miceli-Barone Mar 5 '13 at 11:51
  • $\begingroup$ What edges would have a Bayesian network with nodes A, B and X? $\endgroup$ – Franco Aug 14 '18 at 16:13

There are four possible two-edge trails from X to Y via Z as shown in this figure:

enter image description here

Source: Probabilistic Graphical Models Principles and Techniques (2009, The MIT Press)

These four relations represent all possible structures in a Bayes Graph(we can extend the graph to contain more than three nodes respectively). We can learn from the book that if Z is given, X and Y are independent in the first three relations(this also meets the requirement in your case: X and Y are mutually independent if we imagine that X and Y contain multiple nodes). But in (d), where three nodes form a v-structure, if Z is given X and Y become dependent.

enter image description here

  • $\begingroup$ Could you explain why X and Y are dependent given Z in (d)? What's the difference between the first three models and the last one? $\endgroup$ – John Jul 6 '19 at 4:25
  • $\begingroup$ @Mark Oh, I thought this answer would be helpful. $\endgroup$ – Lerner Zhang Jul 6 '19 at 12:03

It depends on your definition of independence. I have seen at least one textbook that used the term "stocastic indepedence" or something like that to refer to the case where 2 variables were independent, and still conditionally independent when conditioned on other variables. Other textbooks seem to imply this added independence when using the term "independent".

A simple example of 2 variables that are independent without being stocastically independent are the (x,y,z) triplets: (1,2,3); (1,3,2); (2,1,3); (2,3,1); (3,1,2); (3,2,1); (1,1,1); (2,2,2); and (3,3,3). Given any value of x (or y or z) the probabilies for y (or z) are 1/3 for each of the values 1,2,3. But if you know x and y (or any pair) then you know the value of z.

  • $\begingroup$ This is not what I wanted to hear, but these examples are compelling, both Greg's and Antonio's. Hey, I'm new here. When somebody provides a complete answer to my question, am I supposed to do something to acknowledge it, besides say thanks? (Thanks, BTW) $\endgroup$ – andrewH Mar 5 '13 at 2:35
  • $\begingroup$ @andrewH You can up-vote useful answers and comments. The best one that answers your question satisfactorily can be "accepted". i.e. click on that tick mark. $\endgroup$ – curious_cat Mar 5 '13 at 7:06

Two events $A$ and $B$ that are stochastically independent (meaning that $P(A\cap B) = P(A)P(B)$ holds need not be conditionally independent given that some event $C$ occurred, that is, it need not be the case that $P((A\cap B) \mid C) = P(A\mid C)P(B\mid C)$ which is the definition of conditional independence of $A$ and $B$ conditioned on $C$. For example, if we take $C$ to be the event $A\cup B$, then since $A\cap B$, $A$, and $B$ all are subsets of $A\cup B$, we have that \begin{align} P((A\cap B) \mid (A\cup B)) &= \frac{P(A\cap B)}{P(A\cup B)}\\ P(A \mid (A\cup B)) &= \frac{P(A)}{P(A\cup B)}\\ P(B \mid (A\cup B)) &= \frac{P(B)}{P(A\cup B)}\\ \end{align} and it should be obvious that

$$\text{since we have assumed that}~P(A\cap B) = P(A)P(B)~ \text{holds,}\\ \text{it cannot possibly be true that}~ \frac{P(A\cap B)}{P(A\cup B)}=\frac{P(A)}{P(A\cup B)}\times\frac{P(B)}{P(A\cup B)}~\text{also holds}.$$

Nitpickers getting ready to object "But it holds when $P(A\cup B)$ equals $1$, doesn't it??" are reminded that $P(A\cup B) = 1-P(A^c\cap B^c)$ cannot equal $1$ for independent events $A$ and $B$ except in the trivial case when at least one of $A$ and $B$ is an event of probability $1$. $\big($Note that $P(A\cup B)=1$ implies that $P(A^c\cap B^c) = 0$ but by independence, $P(A^c\cap B^c)=P(A^c)P(B^c)$ equals $0$ and so at least one of $A^c$ and $B^c$ has probability $0$ and its complement has probability $1.\big)$ So

Independent events need not be conditionally independent.

But of course there exist conditioning events $C$ such that independent events $A$ and $B$ are also conditionally independent given $C$. Trivially, if $A,B,C$ are stochastically mutually independent events, then $A$ and $B$ are also conditionally independent events conditioned on the occurrence of $C$.

Turning to random variables, if $X$ and $Y$ are independent Bernoulli$\left(\frac 12\right)$ random variables and $Z$ is a Bernoulli random variable that has value $1$ if and only if $X \neq Y$, then $X$ and $Z$ are a pair of independent Bernoulli$\left(\frac 12\right)$ random variables as are $Y$ and $Z$ a pair of independent Bernoulli$\left(\frac 12\right)$ random variables. But, the conditional joint pmf of $X$ and $Y$ given $Z$ does not factor into the product of the conditional marginal pdfs of $X$ and $Y$ showing that $X $and $Y$ are not conditionally independent random variables given $Z$. (This is essentially Antonio's example). But note also that the independence of $X$ and $Z$ and also of $Y$ and $Z$ is stochastic independence that disappears if we assume that $X$ and $Y$ are independent random Bernoulli random variables with parameter $p \notin \{0,\frac 12, 1\}$. So, if one is estimating probabilities of random variables and using these estimates to determine whether it is reasonable to assume stochastic independence or not, then such an approach if fraught with peril. While one might not reject the hypothesis that $p = \frac 12$, not rejecting the null is not the same as a whole-hearted acceptance of the null as the gospel truth.

See also this earlier answer of mine on this topic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.