Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When studying covariance selection, I once read the following example. With respect to the following model:
Its covariance matrix and inverse covariance matrix are given as follows,
I do not understand why the independence of $x$ and $y$ is decided by the inverse covariance here?
What is the mathematical logic underlying this relationship?
Also, the left graph in the following figure is claimed to capture the independence relationship between $x$ and $y$; why?
The inverse covariance matrix can be used to work out conditional variances and covariances for multivariate Gaussian distributions. An earlier question gives some references
For example to find the conditional covariance of $Y$ and $Z$ given the value $X=x$, you would take the bottom right corner of the inverse covariance matrix
$$\left( \begin{array}{rr} 1 & -1 \\ -1 & 3 \end{array} \right) \text{ and re-invert it to }\left( \begin{array}{rr} \tfrac32 & \tfrac12 \\ \tfrac12 & \tfrac12 \end{array} \right)$$
which does indeed give the covariance matrix of $Y$ and $Z$ conditioned on the the value for $X=x$.
So similarly to find the conditional covariance matrix of $X$ and $Y$ given the value for $Z=z$, you would take the top left corner of the inverse covariance matrix
$$\left( \begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array} \right) \text{ and re-invert it to }\left( \begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array} \right)$$
telling you that the conditional covariance between $X$ and $Y$ given $Z=z$ is $0$ (and that each of their conditional variances is $1$).
To conclude that this zero conditional covariance implies conditional independence, you also have to use the fact this is a multivariate Gaussian (as in general zero covariance does not necessarily imply independence). You know this from the construction.
Arguably you also know about the conditional independence from the construction, since you are told that $\epsilon_1$ and $\epsilon_2$ are iid, so conditioned on a particular value for $Z=z$, $X=z+\epsilon_1$ and $Y=z+\epsilon_2$ are also iid. If you know $Z=z$, there is no additional information from $X$ that helps you say anything about possible values of $Y$.
This is a supplement to the correct and accepted answer. In particular, the original question contains a follow-up question about the statement the book makes.
Also, the left graph in the following figure is claimed to capture the independence relationship between $X$ and $Y$, why?
This is what is addressed in this answer, and is the only thing addressed in this answer.
To make sure we are on the same page, in what follows I use this definition of (undirected) conditional independence graph which corresponds (at least roughly) to Markov random fields:
Definition: The conditional independence graph of $X$ is the undirected graph $G=(K,E)$ where $K=\{ 1, 2, \dots, k \}$ and $(i,j)$ is not in the edge set if and only if $X_i \perp \! \! \! \perp X_j | X_{K \setminus \{i,j\}}$. (Where $X_{K \setminus \{i,j\}}$ denotes the vector of all of the random variables except for $X_i$ and $X_j$.)
From p. 60 of Whittaker, Graphical Models in Applied Mathematical Multivariate Statistics (1990).
Here, using the argument given by Henry in the correct, accepted answer, we can establish that $X$ and $Y$ are conditionally independent given $Z$, in notation, $X \perp \! \! \perp Y \ | Z$.
Since the only three random variables are $X, Y$, and $Z$, this means that $X$ and $Y$ are conditionally independent when given all of the other remaining random variables (in this case just $Z$).
Using the definition of conditional independence graph given above, this means that all edges in the graph should be included except for the edge between $X$ and $Y$. Indeed, this is exactly what is shown on the right graph of that picture.
Regarding the left graph, it is unclear without having more context, but I think the idea is just to show what the conditional independence graph would look like if we didn't have zeros in those entries of the inverse covariance matrix.
In particular, using the above definition, we see that we can start with the complete graph on the nodes $X, Y, Z$, which is the left graph in that picture, and then derive the conditional independence graph from that first graph by removing all edges corresponding to conditionally independent random variables. The picture does compare the two graphs explicitly ("versus"), which to me suggests a comparison between the complete graph one might start with and the conditional independence graph one ends up with if/when they apply the definition of conditional independence graph as given above.
