Intuitive Understanding of the Effect of Correlation on Random Variables in Gaussian Process

So In general covariance matrix in GP provides us with proportionality relation between random variables, in other words $$x_1$$ and $$x_2$$ are perfectly correlated if off-diagonal entry has $$\rho=\pm 1$$: $$\begin{bmatrix} \sigma_x^2 & 1*\sigma_y\sigma_x\\ 1*\sigma_x\sigma_y & \sigma_y^2 \end{bmatrix}$$

so far so good. Now if we construct the plot with 0 $$\mu$$ and unit variance/covariance, and take 5 samples, it shall look something like this:

So this makes sense, they both are correlated exactly so they are same. Now with this same "understanding" for three random variables we can have the following three different kind of $$\Sigma$$ assuming unit variance and $$\rho= 1$$: $$\begin{bmatrix} 1 & 1& 1\\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}, \begin{bmatrix} 1 & 1& 0\\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \begin{bmatrix} 1 & 0& 1\\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix}$$ So from left - right: in the first all variables are correlated, in second only $$x_1,x_2$$ are correlated and in the third only $$x_1,x_3$$ are correlated. So technically in the first matrix knowing the value of say $$x_1$$ should determine the value of $$x_2,x_3$$ as well, however if we include corresponding values of each ramdom variables in the mean of GP, the correlation has "no effect" at all. The following is the plot for three covariance matrices with $$\mu(x_1)$$ = 10 $$\mu(x_2)$$ = 0, $$\mu(x_3)$$ = 5 and the question is:

What role do covariance matrix play in this case?

:

import numpy as np
from matplotlib import pyplot as plt
# Finite number of x points
X = [0,1,2]
# Finite number of x points
samples1 = np.random.multivariate_normal([10,0,5], [[1,1,1],[1,1,1],[1,1,1]],5)
samples2 = np.random.multivariate_normal([10,0,5], [[1,0,1],[0,1,0],[1,0,1]],5)
samples3 = np.random.multivariate_normal([10,0,5], [[1,1,0],[1,1,0],[0,0,1]],5)

plt.figure()
for i in range(len(samples1)):
plt.plot(X, samples1[i],'-o')
plt.title("Σ=[[1,1,1],[1,1,1],[1,1,1]]")
plt.figure()
for i in range(len(samples2)):
plt.plot(X, samples2[i],'-o')
plt.title("Σ=[[1,0,1],[0,1,0],[1,0,1]]")
plt.figure()
for i in range(len(samples3)):
plt.plot(X, samples3[i],'-o')
plt.title("Σ=[[1,1,0],[1,1,0],[0,0,1]]")
plt.show()



Let me try and help with the question. I think the best way to understand a Multivariate distribution is less as random variables, but instead as random vectors in $$\mathbb{R}^n$$. I may be wrong but I get the sense that you are thinking that $$X_1, X_2, ..$$ are drawn separately. Instead think of the vector as produced from the sample space ($$\Omega \to \mathbb{R^n}$$). Its hard to see in 3-d but what the sample function does is repeat this process. Here it is in 2-d with different cross correlation coefficients. I am not drawing $$X_1$$ and then drawing $$X_2$$ separately they both come together and land on the $$\mathbb{R}^2$$ plane.

The nice thing about Gaussians is the marginalization. So if I throw away my $$X_2$$ or $$X_1$$ and just focus on the on one variable, think of this as projecting to the x or y axis, then I get a distribution in one dimension, when I plot the histogram for each $$X$$ variable. As you can see I get a familiar looking bell shape centered at 0, despite the correlation being different in the joint distribution for each $$X$$.

But as you say knowing $$X_1$$ should help me to determine the value of $$X_2$$ which is sort of reversing the projection. I know where I am on the $$X_1$$ line can I get a distribution over my X_2. Clearly yes you can, but you need the conditional distribution $$p(X_2|X_1)$$. You can think of the distribution as drawing a line vertically and looking at points that lie on it. Clearly to do that you need the correlation between the two which if you look on wiki pops up in the calculation.

So in answer to the question perhaps don’t think about the covariance matrix think about the conditional distribution which is key in GPs. Looking at your charts they fit with intuition. Specifically the second plot I think shows that linear relationship between $$X_1$$ and $$X_2$$ but $$X_3$$ having some stochasticity around the mean. But the covariance of $$X_3$$ to the other two is 0 so you are essentially just drawing a unit variance Gaussian around its mean.

• +1 for great explanation! I am still little short of catching the intuition. So taking concrete example of the middle plot that you took, you mentioned in that plot $X_3$ is just drawing a unit variance around its mean. So now the question is: Are $X_1$ and $X_2$ drawing "not" drawing a unit variance around its mean in that case? Second thing is that I do see linear relationship but I dont understand in what sense they are linear? The lines are parallel between $X_1$ and $X_2$ but what does it mean, so taking your explanation what does $p(X_2|X_1)$ is in that case of the middle plot? – GENIVI-LEARNER Feb 19 '20 at 15:14
• I am quite keen on knowing what is $p(X_2|X1)$ in the case of middle plot. – GENIVI-LEARNER Feb 20 '20 at 16:18
• I think my point is that its not really that useful to have a correlations of 1. You no longer have two random variables just one. So no point in computing the conditional. As you can see they both miss their mean by the same amount. Strictly speaking you shouldn't be able to plot this as you have a less than full rank covariance matrix, so you dont have a joint density defined. A line in Lebesgue measure is 0. – Chango Feb 21 '20 at 9:17
• yes got it. So saying $p(X_2|X_1)$ is same like saying $p(X_2|X_2)$ and they both are $p(X_2|X_1)$=$p(X_2|X_2)$=1. right? – GENIVI-LEARNER Feb 21 '20 at 12:55
• Yeah sort of. $\mu_1|\mu_2 = \mu_1 + (x_2 - \mu_2)$ and its conditional variance is 0. So $p(X_1|X_2) = N(\mu_1 + (x_2 - \mu_2),0)$. So by having a conditional variance of 0, if you know $X_2$ you know exactly $X_1$ and vice versa. – Chango Feb 21 '20 at 13:14