Conditional independence isn't guaranteed when specifying the marginal distributions?

Question

It was mentioned very briefly in a lecture related to graphical models that two random variables $X_3$ and $X_4$ are both dependent on $X_2$. But even when conditioned on $X_2$, the two variables $X_3$ and $X_4$ could be dependent through some other means.

I wasn't sure what the person meant by this, below is an example provided:

Random variables: $X_1$, $X_2$, $X_3$, $X_4$ with the following density functions.

$X_1 \sim \mathrm{Bernoulli}(1/2)$
$X_2 \sim \mathrm{Bernoulli}(1/2)$
$X_3 \mid (X_1,X_2) \sim \mathcal N(X_1+X_2,\sigma^2)$
$X_4 \mid X_2 \sim \mathcal N(aX_2+b,1)$

The above specification doesn't necessarily mean that:

$p(x_3,x_4|x_2)=p(x_3|x_2)p(x_4|x_2)$

Meaning the conditional independence does not necessarily hold between $X_3$ and $X_4$ given $X_2$. The graphical model shows that the conditional independence will hold, but the specification I provided above does not guarantee it.

Answer 1

This question and the OP's lecturer's claims seem to indicate misunderstanding of the notions of independence and conditional independence of random variables. Different sets of distributions for Bernoulli random variables $X$, $Y$, and $Z$ are presented here to illustrate the differences between various notions.

• Suppose that $X$ and $Y$ are known to be independent Bernoulli random variables with parameter $\frac{1}{2}$. Thus, their probability mass functions (pmf) are $$p_X(0) = p_X(1) = \frac{1}{2}; ~ p_Y(0) = p_Y(1) = \frac{1}{2}$$ and their joint pmf is the product of the (marginal) pmfs $$p_{X,Y}(i,j) = p_X(i)p_Y(j) = \frac{1}{2}\times\frac{1}{2} = \frac{1}{4} ~\text{for all} ~ i, j \in \{0, 1\}$$ Are $X$ and $Y$ necessarily conditionally independent given $Z$ where $Z$ is also a Bernoulli random variable with parameter $\frac{1}{2}$? Not necessarily. Suppose that $Z$ has parameter $\frac{1}{2}$ and consider the probability distributions $$\begin{align*}p_{X,Y\mid Z}(i,j\mid Z=0) &= \begin{cases}\frac{1}{2}, & i = j\\ 0, & i \neq j, \end{cases}\\ p_{X,Y\mid Z}(i,j\mid Z=1) &= \begin{cases}\frac{1}{2}, & i \neq j\\ 0, & i = j, \end{cases} \end{align*} $$ The law of total probability shows that these conditional distributions combine to give the known joint pmf of $X$ and $Y$. It is also easy to verify that regardless of whether $Z = 0$ or $Z = 1$, both $X$ and $Y$ are conditionally distributed as Bernoulli random variables with parameter $\frac{1}{2}$, but $X$ and $Y$ are not conditionally independent given $Z$, regardless of whether $Z$ has value $0$ or $1$. In one case, we have $X = Y$, and in the other, $X = 1-Y$. Thus we can say the following

If $X$ and $Y$ are independent random variables with known marginal distributions, then their joint distribution is the product of the marginal distributions. However, $X$ and $Y$ need not be conditionally independent even if the conditional marginal distributions of $X$ and $Y$ are the same as their given unconditional marginal distributions. Thus the conditional joint distribution of unconditionally independent random variables need not be the product of the conditional marginal distributions.

• Suppose that $X$ and $Y$ are conditionally independent given $Z = 0$ and also conditionally independent given $Z = 1$. Are $X$ and $Y$ necessarily unconditionally independent? Not necessarily, not even if $Z$ is a Bernoulli random variable with parameter $\frac{1}{2}$. Suppose that $X$ and $Y$ are conditionally independent Bernoulli random variables with parameter $p$ if $Z = 0$ and parameter $q$ if $Z = 1$. Thus, the conditional joint pmfs are $$\begin{align*} p_{X,Y\mid Z}(0,0\mid Z = 0) &= (1-p)^2; \qquad \quad p_{X,Y\mid Z}(0,0\mid Z = 1) = (1-q)^2;\\ p_{X,Y\mid Z}(0,1\mid Z = 0) &= p(1-p); \qquad \quad p_{X,Y\mid Z}(0,1\mid Z = 1) = q(1-q);\\ p_{X,Y\mid Z}(1,0\mid Z = 0) &= p(1-p); \qquad \quad p_{X,Y\mid Z}(1,0\mid Z = 1) = q(1-q);\\ p_{X,Y\mid Z}(1,1\mid Z = 0) &= p^2; \qquad \quad \, \qquad p_{X,Y\mid Z}(1,1\mid Z = 1) = q^2; \end{align*}$$ Suppose that $p \neq q$. Then, $X$ and $Y$ are unconditionally independent only in the trivial cases when $Z$ has parameter $\lambda$ equal to $0$ or $1$ (when one of the above two joint pmfs has weight $0$ in the total probability formula.

If $X$ and $Y$ are conditionally independent given $Z$, they need not be unconditionally independent.

Is there any instance where conditional independence guarantees unconditional independence? If $X$ and $Y$ are not only conditionally independent given $Z$ but also have the same conditional joint distribution for all choices of $Z$ then $X$ and $Y$ are unconditionally independent. But this is also a trivial special case because the necessary condition means that $X$, $Y$, and $Z$ are mutually independent random variables, and so the conditional joint distribution of $X$ and $Y$ does not depend on the value of $Z$.

Answer 2

The following is a long and argumentative answer with the first part being somewhat philosophical and the second part being most relevant to the question asked here.

In probability theory, independence is usually an assumption that is given as part of the problem statement as in "Let $A$ and $B$ denote independent events" or "Let $X$ and $Y$ denote independent random variables" and this means that the events or random variables enjoy a property that does not hold for events or random variables in general. Specifically, $$P(A\cap B) = P(A)P(B)$$ and $$P\{X \in \mathcal A, Y \in \mathcal B\} =P\{X \in \mathcal A\}P\{Y \in \mathcal B\}$$ for all sets $\mathcal A$ and $\mathcal B$ in the $\sigma$-algebra. Independence of random variables $X$ and $Y$ implies the more commonly used statement $$P\{X \leq a, Y \leq b\} = F_{X,Y}(a,b) = P\{X \leq a\}P\{Y \leq b\} = F_{X}(a)F_{Y}(b), \forall a, b$$ as well as $p_{X,Y}(a,b) = p_X(a)p_Y(b)~ \forall a, b$ for discrete random variables and $f_{X,Y}(a,b) = f_X(a)f_Y(b)~\forall a, b$ for continuous random variables (note that independence of continuous random variables $X$ and $Y$ implies joint continuity of $X$ and $Y$). Let us call this the product rule.

In practical applications, where $X$ and $Y$ are random variables modeling physical phenomena, things are not quite so straightforward. If $X$ and $Y$ are arising from distinct physically unrelated sources, we take them to be independent and apply the product rule. Note that there is no proof of independence, only a general feeling that one phenomenon does not affect the other and so independence is a reasonable assumption. In some cases, of course, beauty might lie in the eye of the beholder (or the author, reviewer, and editor) but might be completely invisible to the general reader. But the notion is that physical independence, whether such independence is assumed or a reasonable argument made in support thereof, justifies the use of the product rule.

A third form of independence is stochastic independence: $A$ and $B$ are said to be independent events if $P(A\cap B) = P(A)P(B)$. Physical independence implies stochastic independence but the converse need not be true. Here is an example whose underlying idea might be familiar to the reader in the guise of coin-tossing.

Consider an Exclusive-OR (XOR) circuit whose inputs $X$ and $Y$ are are physically independent Bernoulli random variables taking on values $0$ and $1$ with probability $\frac{1}{2}$. Physically independent because they are coming from physically independent sources, say from two different data packets sent by two different computers in two different parts of the country. The output $Z = X \oplus Y$ has value $1$ exactly when one of $X$ and $Y$ has value $1$ and the other has value $0$. It is easy to show that $Z$ is also a Bernoulli random variable with parameter $\frac{1}{2}$. Now, $Z$ is clearly physically dependent on $X$ -- the output of an XOR circuit should depend on its inputs -- but $Z$ and $X$ are stochastically independent random variables: $$\begin{align*} p_{X,Z}(0,0) &= P\{X = 0, Z = 0\} = P\{X = 0, Y = 0\} = p_{X,Y}(0,0) = \frac{1}{4} = p_X(0)p_Z(0)\\ p_{X,Z}(0,1) &= P\{X = 0, Z = 1\} = P\{X = 0, Y = 1\} = p_{X,Y}(0,1) = \frac{1}{4} = p_X(0)p_Z(1)\\ p_{X,Z}(1,0) &= P\{X = 1, Z = 0\} = P\{X = 1, Y = 1\} = p_{X,Y}(1,1) = \frac{1}{4} = p_X(1)p_Z(0) \\p_{X,Z}(1,1) &= P\{X = 1, Z = 1\} = P\{X = 1, Y = 0\} = p_{X,Y}(1,0) = \frac{1}{4} = p_X(1)p_Z(1) \end{align*}$$ Similarly, $Y$ and $Z$ are stochastically independent but physically dependent random variables.

Thus, stochastic independence does not necessarily mean physical independence; it might just be an artifact of the probability assignment. If we re-work the above example with $X$ and $Y$ physically independent Bernoulli random variables with parameter $p \in (0,1), ~ p \neq \frac{1}{2}$, then we see that $Z$ is no longer stochastically independent of $X$. Thus, the independence of $X$ and $Z$ is an artifact of the probability assignment or the possibility that when the model was being devised, the hypothesis $p = \frac{1}{2}$ did not get rejected because the test did not give a statistically significant result!

---

---

Currently studies that can be tarred broadly as saying "Everything is independent unless someone beats me on the head and says it is not." seem to be in great vogue.

---

---

Turning to the OP's edited question,

Random variables: $X_1$, $X_2$, $X_3$, $X_4$ with the following density functions.

$X_1 \sim \mathrm{Bernoulli}(1/2)$
$X_2 \sim \mathrm{Bernoulli}(1/2)$
$X_3 \mid (X_1,X_2) \sim \mathcal N(X_1+X_2,\sigma^2)$
$X_4 \mid X_2 \sim \mathcal N(aX_2+b,1)$

The above specification doesn't necessarily mean that:

$p(x_3,x_4|x_2)=p(x_3|x_2)p(x_4|x_2)$

Note that nothing is said about whether $X_1$ and $X_2$ are independent, whether physically or stochastically, nor is anything said about independence or lack thereof of $X_3$ and $X_4$. Are they all assumed to be independent because everything is independent?

The conditional joint density of $X_3$ and $X_4$ given $X_2$ cannot be computed from the given information, and so whether or not the product rule $$p(x_3,x_4|x_2)=p(x_3|x_2)p(x_4|x_2)$$ holds or not cannot be determined from the given information. In any case, there seems to be no obvious reason why the product rule should hold, and I don't see the faiure as being indicative of anything except that the assumption of independence everywhere does not seem to be working here.