2
$\begingroup$

Let $Z_i \sim \mathcal{N}(0,1)$ be independent normal distributions. Consider the following correlated variables, defined by $$ X_1 = \frac{Z_1 + Z_2}{\sqrt{2}},\;\;\;X_2= \frac{Z_2 + Z_3}{\sqrt{2}},\;\;\;X_3= \frac{Z_3 + Z_4}{\sqrt{2}},\ldots$$

Thus each $X_i$ by itself is also a standard normal distribution but is correlated to the immediate neighbours $X_{i-1}$ and $X_{i+1}$. Consider the joint distribution of $(X_1,X_2)$ which is a joint normal with mean = $(0,0)$ and covariance matrix $$\begin{pmatrix} 1 & 1/2 \\ 1/2 & 1 \end{pmatrix} $$

Now the thing is according to the rules of conditional probability the conditional variance for $X_1$ is $\left(1-\rho^2\right)\sigma_1^2 = \frac{3}{4}$ in this case. So far so good.

Suppose we then consider the joint normal $\left(X_1,X_2,X_3\right)$, which has the covariance matrix $$\begin{pmatrix} 1 & 1/2 & 0 \\ 1/2 & 1 & 1/2 \\ 0 & 1/2 & 1 \end{pmatrix}$$

In this case, the conditional variance of $X_1$ is given by

$$ 1 - \begin{pmatrix} 1/2 & 0 \end{pmatrix}\begin{pmatrix} 1 & 1/2 \\ 1/2 & 1 \end{pmatrix}^{-1}\begin{pmatrix} 1/2 \\ 0 \end{pmatrix} = 2/3$$

The questions I have now are:

  1. Since $X_1$ is not dependent on $X_3$ at all, why the conditional variance of $X_1$ drops from $3/4$ to $2/3$ when $X_3$ is taken into account?
  2. If I further include $X_4,X_5,\ldots$ the conditional variance seems to drop further and reaches a limit of $1/2$ when I include a very large number of $X_i$. Is there any intuitive explanation for this limit?
$\endgroup$
  • $\begingroup$ Because $X_3$ provides some information about $Z_3$ and hence about $X_2$? Work backwards using this notion when you know the values of large numbers of $X_k$. $\endgroup$ – Dilip Sarwate Oct 7 '16 at 1:02
1
$\begingroup$

This is a super interesting question/observation.... First thing to note is that $$Var(X_1\mid Z_2) = 1/2$$

And the reason that $$Var(X_1\mid X_2) = 3/4 \neq 1/2$$

is because it's not possible to disentangle the actual value that $Z_2$ takes when you only observe $$X_2 = \frac{Z_2 + Z_3}{\sqrt{2}}$$

and as the first commentor pointed out, observing $X_3, X_4, X_5, \dots$ tells you more and more information about what the actual value of $Z_2$ is, and thus brings you closer to that $1/2$ bound you mentioned.

To prove the we eventually hit that $1/2$ limit, denote the conditional variance of $X_1$ on $X_2, \dots, X_T$ by $\overline{\Sigma}_T$, and the unconditional covariance matrix of $X_1, X_2, \dots , X_T$ by $\Sigma_T$. We know that $$\overline{\Sigma}_T = \Sigma^{11}_T - \Sigma^{12}_T[\Sigma^{22}_T]^{-1}\Sigma^{21}_T $$

Where we've partitioned the unconditional covariance matrix $\Sigma_T$ as $$\Sigma_T=\begin{bmatrix} \Sigma^{11}_T & \Sigma^{12}_T\\ \Sigma^{21}_T & \Sigma^{22}_T \end{bmatrix} $$

We know that, in this case, $$\Sigma^{11}_T = 1$$ $$\Sigma^{12}_T = [1/2, \boldsymbol{0}]$$ $$\Sigma^{21}_T = [1/2, \boldsymbol{0}]^T$$ where $\boldsymbol{0}$ is an 1x(T-1) vectors of zeros (this is because $X_1$ is dependent only on $X_2$ and not on $X_3, X_4, \dots$).

Now, we can write $$[\Sigma^{22}_T]^{-1} = \begin{bmatrix}\Sigma^{11}_{T-1} & \Sigma^{12}_{T-1}\\ \Sigma^{21}_{T-1} & \Sigma^{22}_{T-1} \end{bmatrix}^{-1} = \begin{bmatrix} (1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1} & (1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1}\Sigma^{12}_{T-1}[\Sigma^{22}_{T-1}]^{-1}\\ \vdots & \vdots \end{bmatrix}$$

Where I left out the last row to save space (I'll show those entries don't matter anyways). This result holds by the Schur complement of a block matrix.

Next, since $$\Sigma^{12}_T = [1/2, \boldsymbol{0}]$$ $$\Sigma^{21}_T = [1/2, \boldsymbol{0}]^T$$ this all simplifies to $$\overline{\Sigma}_T = 1 - \frac{1}{4}(1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1})^{-1}$$

However, since $\overline{\Sigma}_{T-1} = 1 - \Sigma^{12}_{T-1} [\Sigma^{22}_{T-1}]^{-1}\Sigma^{21}_{T-1}$, we see finally that

$$\overline{\Sigma}_T = 1 - \frac{1}{4}[\overline{\Sigma}_{T-1}]^{-1}$$

Taking the limit as $T\rightarrow \infty$ yields $$\overline{\Sigma}_{\infty} = 1 - \frac{1}{4}[\overline{\Sigma}_{\infty}]^{-1}$$

Which can be solved to give the limiting conditional variance as $$\overline{\Sigma}_\infty = 0.5$$

$\endgroup$
  • $\begingroup$ I really like your answer to the second part. $\endgroup$ – hermitian Oct 10 '16 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.