# Peculiar Behaviour of Conditional Variance for Multivariate Normal Distributions

Let $Z_i \sim \mathcal{N}(0,1)$ be independent normal distributions. Consider the following correlated variables, defined by $$X_1 = \frac{Z_1 + Z_2}{\sqrt{2}},\;\;\;X_2= \frac{Z_2 + Z_3}{\sqrt{2}},\;\;\;X_3= \frac{Z_3 + Z_4}{\sqrt{2}},\ldots$$

Thus each $X_i$ by itself is also a standard normal distribution but is correlated to the immediate neighbours $X_{i-1}$ and $X_{i+1}$. Consider the joint distribution of $(X_1,X_2)$ which is a joint normal with mean = $(0,0)$ and covariance matrix $$\begin{pmatrix} 1 & 1/2 \\ 1/2 & 1 \end{pmatrix}$$

Now the thing is according to the rules of conditional probability the conditional variance for $X_1$ is $\left(1-\rho^2\right)\sigma_1^2 = \frac{3}{4}$ in this case. So far so good.

Suppose we then consider the joint normal $\left(X_1,X_2,X_3\right)$, which has the covariance matrix $$\begin{pmatrix} 1 & 1/2 & 0 \\ 1/2 & 1 & 1/2 \\ 0 & 1/2 & 1 \end{pmatrix}$$

In this case, the conditional variance of $X_1$ is given by

$$1 - \begin{pmatrix} 1/2 & 0 \end{pmatrix}\begin{pmatrix} 1 & 1/2 \\ 1/2 & 1 \end{pmatrix}^{-1}\begin{pmatrix} 1/2 \\ 0 \end{pmatrix} = 2/3$$

The questions I have now are:

1. Since $X_1$ is not dependent on $X_3$ at all, why the conditional variance of $X_1$ drops from $3/4$ to $2/3$ when $X_3$ is taken into account?
2. If I further include $X_4,X_5,\ldots$ the conditional variance seems to drop further and reaches a limit of $1/2$ when I include a very large number of $X_i$. Is there any intuitive explanation for this limit?
• Because $X_3$ provides some information about $Z_3$ and hence about $X_2$? Work backwards using this notion when you know the values of large numbers of $X_k$. – Dilip Sarwate Oct 7 '16 at 1:02

This is a super interesting question/observation.... First thing to note is that $$Var(X_1\mid Z_2) = 1/2$$

And the reason that $$Var(X_1\mid X_2) = 3/4 \neq 1/2$$

is because it's not possible to disentangle the actual value that $Z_2$ takes when you only observe $$X_2 = \frac{Z_2 + Z_3}{\sqrt{2}}$$

and as the first commentor pointed out, observing $X_3, X_4, X_5, \dots$ tells you more and more information about what the actual value of $Z_2$ is, and thus brings you closer to that $1/2$ bound you mentioned.

To prove the we eventually hit that $1/2$ limit, denote the conditional variance of $X_1$ on $X_2, \dots, X_T$ by $\overline{\Sigma}_T$, and the unconditional covariance matrix of $X_1, X_2, \dots , X_T$ by $\Sigma_T$. We know that $$\overline{\Sigma}_T = \Sigma^{11}_T - \Sigma^{12}_T[\Sigma^{22}_T]^{-1}\Sigma^{21}_T$$

Where we've partitioned the unconditional covariance matrix $\Sigma_T$ as $$\Sigma_T=\begin{bmatrix} \Sigma^{11}_T & \Sigma^{12}_T\\ \Sigma^{21}_T & \Sigma^{22}_T \end{bmatrix}$$

We know that, in this case, $$\Sigma^{11}_T = 1$$ $$\Sigma^{12}_T = [1/2, \boldsymbol{0}]$$ $$\Sigma^{21}_T = [1/2, \boldsymbol{0}]^T$$ where $\boldsymbol{0}$ is an 1x(T-1) vectors of zeros (this is because $X_1$ is dependent only on $X_2$ and not on $X_3, X_4, \dots$).

Now, we can write $$[\Sigma^{22}_T]^{-1} = \begin{bmatrix}\Sigma^{11}_{T-1} & \Sigma^{12}_{T-1}\\ \Sigma^{21}_{T-1} & \Sigma^{22}_{T-1} \end{bmatrix}^{-1} = \begin{bmatrix} (1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1} & (1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1}\Sigma^{12}_{T-1}[\Sigma^{22}_{T-1}]^{-1}\\ \vdots & \vdots \end{bmatrix}$$

Where I left out the last row to save space (I'll show those entries don't matter anyways). This result holds by the Schur complement of a block matrix.

Next, since $$\Sigma^{12}_T = [1/2, \boldsymbol{0}]$$ $$\Sigma^{21}_T = [1/2, \boldsymbol{0}]^T$$ this all simplifies to $$\overline{\Sigma}_T = 1 - \frac{1}{4}(1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1}$$

However, since $\overline{\Sigma}_{T-1} = 1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1}$, we see finally that

$$\overline{\Sigma}_T = 1 - \frac{1}{4}[\overline{\Sigma}_{T-1}]^{-1}$$

Taking the limit as $T\rightarrow \infty$ yields $$\overline{\Sigma}_{\infty} = 1 - \frac{1}{4}[\overline{\Sigma}_{\infty}]^{-1}$$

Which can be solved to give the limiting conditional variance as $$\overline{\Sigma}_\infty = 0.5$$

• I really like your answer to the second part. – hermitian Oct 10 '16 at 11:22