Intuitive Understanding of the Effect of Correlation on Random Variables in Gaussian Process

Question

So In general covariance matrix in GP provides us with proportionality relation between random variables, in other words $x_1$ and $x_2$ are perfectly correlated if off-diagonal entry has $\rho=\pm 1$: $$\begin{bmatrix} \sigma_x^2 & 1*\sigma_y\sigma_x\\ 1*\sigma_x\sigma_y & \sigma_y^2 \end{bmatrix}$$

so far so good. Now if we construct the plot with 0 $\mu$ and unit variance/covariance, and take 5 samples, it shall look something like this:

So this makes sense, they both are correlated exactly so they are same. Now with this same "understanding" for three random variables we can have the following three different kind of $\Sigma$ assuming unit variance and $\rho= 1$: $$\begin{bmatrix} 1 & 1& 1\\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}, \begin{bmatrix} 1 & 1& 0\\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \begin{bmatrix} 1 & 0& 1\\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix}$$ So from left - right: in the first all variables are correlated, in second only $x_1,x_2$ are correlated and in the third only $x_1,x_3$ are correlated. So technically in the first matrix knowing the value of say $x_1$ should determine the value of $x_2,x_3$ as well, however if we include corresponding values of each ramdom variables in the mean of GP, the correlation has "no effect" at all. The following is the plot for three covariance matrices with $\mu(x_1)$ = 10 $\mu(x_2)$ = 0, $\mu(x_3)$ = 5 and the question is:

What role do covariance matrix play in this case?

:

import numpy as np
from matplotlib import pyplot as plt
# Finite number of x points
X = [0,1,2]
# Finite number of x points
samples1 = np.random.multivariate_normal([10,0,5], [[1,1,1],[1,1,1],[1,1,1]],5)
samples2 = np.random.multivariate_normal([10,0,5], [[1,0,1],[0,1,0],[1,0,1]],5)
samples3 = np.random.multivariate_normal([10,0,5], [[1,1,0],[1,1,0],[0,0,1]],5)

plt.figure()
for i in range(len(samples1)):
    plt.plot(X, samples1[i],'-o')
    plt.title("Σ=[[1,1,1],[1,1,1],[1,1,1]]")
plt.figure()
for i in range(len(samples2)):
    plt.plot(X, samples2[i],'-o')
    plt.title("Σ=[[1,0,1],[0,1,0],[1,0,1]]")
plt.figure()
for i in range(len(samples3)):
    plt.plot(X, samples3[i],'-o')
    plt.title("Σ=[[1,1,0],[1,1,0],[0,0,1]]")
plt.show()

Answer 1

Let me try and help with the question. I think the best way to understand a Multivariate distribution is less as random variables, but instead as random vectors in $\mathbb{R}^n$. I may be wrong but I get the sense that you are thinking that $X_1, X_2, ..$ are drawn separately. Instead think of the vector as produced from the sample space ($\Omega \to \mathbb{R^n}$). Its hard to see in 3-d but what the sample function does is repeat this process. Here it is in 2-d with different cross correlation coefficients. I am not drawing $X_1$ and then drawing $X_2$ separately they both come together and land on the $\mathbb{R}^2$ plane.

The nice thing about Gaussians is the marginalization. So if I throw away my $X_2$ or $X_1$ and just focus on the on one variable, think of this as projecting to the x or y axis, then I get a distribution in one dimension, when I plot the histogram for each $X$ variable. As you can see I get a familiar looking bell shape centered at 0, despite the correlation being different in the joint distribution for each $X$.

But as you say knowing $X_1$ should help me to determine the value of $X_2$ which is sort of reversing the projection. I know where I am on the $X_1$ line can I get a distribution over my X_2. Clearly yes you can, but you need the conditional distribution $p(X_2|X_1)$. You can think of the distribution as drawing a line vertically and looking at points that lie on it. Clearly to do that you need the correlation between the two which if you look on wiki pops up in the calculation.

So in answer to the question perhaps don’t think about the covariance matrix think about the conditional distribution which is key in GPs. Looking at your charts they fit with intuition. Specifically the second plot I think shows that linear relationship between $X_1$ and $X_2$ but $X_3$ having some stochasticity around the mean. But the covariance of $X_3$ to the other two is 0 so you are essentially just drawing a unit variance Gaussian around its mean.