I am trying to understand what it means for random variables to be (unconditionally) independent but conditionally dependent. Let $(X,Y,Z)$ be a set of three random variables. I understand the usual example of coin flips: $X$ is Bernoulli representing one coin, $Y$ is Bernoulli representing another coin, and $Z = X+ Y$. Here, knowledge of the values of $Z$ and $X$ help in the determination of the value of $Y$ (likewise, $Z$ and $Y$ for $X$).

However, I am confused about Gaussian random variables. Let the set $(X,Y,Z)$ now be a set of jointly Gaussian random variables. Let $Z=X+Y+N(0,1)$. Usually, independence and conditional independence for Gaussians are assessed by correlation and partial correlation, respectively.

$X$ and $Y$ are independent iff the correlation coefficient between $X$ and $Y$ is $r=0$. Similarly, $X$ and $Y$ are independent given $Z$ iff the partial correlation coefficient between $X$ and $Y$ is $r \not = 0$. What exactly is happening that is making the partial correlation coefficient become non-zero here? In the partial correlation test, we regress out $Z$ from $X$ and $Y$ separately, so I don't understand how $r$ becomes non-zero between $X$ and $Y$...


1 Answer 1


First of all, don't ever write things like $Z = X+Y+N(0,1)$, not in public as on this forum, not on your exams, and not even in your notebook or private diary: in fact, don't even think of $N(0,1)$ as being the same as a standard normal random variable. Second, disabuse yourself of the notion that "$X$ and $Y$ are independent iff the correlation coefficient between $X$ and $Y$ is $r=0$." This assertion is not always true, not even when $X$ and $Y$ are restricted to be normal random variables. (It is true if we restrict $X$ and $Y$ to be jointly normal random variables; the presence of that adjective jointly is crucial here).

Turning to your question, you seem to be starting from three independent normal random variables $X, Y, W$ and then considering the three variables $X, Y$, and $Z = X+Y+W$. Since the vector $(X,Y,Z)$ is obtained via a linear transformation from the jointly normal vector $(X,Y,W)$, $X, Y$, and $Z$ also have a jointly normal distribution. Note, though, that while (by assumption) $X, Y, W$ are mutually independent random variables, $X, Y$, and $Z$ are not mutually independent. In fact, both $X$ and $Y$ have nonzero correlation with $Z$. Specifically,

\begin{align} \operatorname{cov}(X,Z) &= \operatorname{cov}(X,X+Y+W) = \operatorname{cov}(X,X) = \operatorname{var}(X)\\ \operatorname{cov}(Y,Z) &= \operatorname{cov}(Y,X+Y+W) = \operatorname{cov}(Y,Y) = \operatorname{var}(Y) \end{align}

Also, $$\mu_Z = \mu_X + \mu_Y + \mu_W, \quad \text{and} \quad\operatorname{var}(Z) = \operatorname{var}(X) + \operatorname{var}(Y) + \operatorname{var}(W).$$

Turning to the joint distribution of $X$ and $Y$ conditioned on $Z = z$, note first that this is a bivariate normal distribution. It is a standard result (see e.g. Wikipedia) that the mean vector of this bivariate distribution is

$$\left[\begin{matrix}\hat{\mu}_X\\ \\ \hat{\mu}_Y \end{matrix}\right] = \left[\begin{matrix}{\mu}_X + \left.\left.\frac{\operatorname{var}(X)}{\operatorname{var}(Z)}\right(z - \mu_Z\right)\\ {\mu}_Y + \left.\left.\frac{\operatorname{var}(Y)}{\operatorname{var}(Z)}\right(z - \mu_Z\right)\end{matrix}\right] $$ while the covariance matrix is $$\left[\begin{matrix}\operatorname{var}(X) & 0\\ 0 & \operatorname{var}(Y)\end{matrix}\right] - \frac{1}{\operatorname{var}(Z)}\left[\begin{matrix}(\operatorname{var}(X))^2 & \operatorname{var}(X)\operatorname{var}(Y)\\ \operatorname{var}(X)\operatorname{var}(Y) & (\operatorname{var}(Y))^2\end{matrix}\right].$$ Notice that the covariance matrix is not a diagonal matrix, that is,

$X$ and $Y$ are independent normal random variables (and thus jointly normal random variables too), but, conditioned on the value of $Z = X+Y+W$, they are conditionally dependent (jointly normal) random variables.

Why is this so? Well, broadly speaking, knowing the value of $Z$ tells us a little bit about the values of $X$ and $Y$, and what is still unknown about $X$ and $Y$ is thus linked through this shared information. Instead of being independent, they are conditionally dependent. Think of $Z$ as a noisy observation of $X+Y$ where the "noise" is $W$. If the observation of $X+Y$were completely noiseless, then it should be obvious that $X$ and $Y$, even though they are independent, cannot possibly be conditionally independent given that $X+Y = z$ because $Y$ must necessarily equal $z-X$, that is, the value of $X$ determines the value of $Y$. Even for noisy observations, we should be skeptical of assertions or insistences that $X$ and $Y$ are conditionally independent given that $X+Y+W = z$, and indeed the math shows that $X$ and $Y$ are conditionally dependent random variables given the value of a noisy observation $X+Y+W$ of their sum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.