Suppose I have a 3D multivariate gaussian, $[X_1,X_2,X_3]$. $var(X_i) = 1, cov(X_2,X_3)=cov(X_1,X_2)=0.98, \mu=0$. As I vary $cov(X_1,X_3)$ from $0.921$ (~ the minimum so that the covariance matrix is positive semidefinite) to ~ 1 (can get as close as we want), a thing happens that is confusing me, and I'm hoping someone here can shed some light.

Suppose that $X_1$ is given as -1 and $X_2$ as 1. The mean/median/mode value of $X_3$ is very different depending on the covariance. As $cov(X_1,X_3)$ goes to 1, the mean/median/mode of $X_3$ ~ -1. This is expected, since $X_1$ was measured at -1, and $X_3$ covaries with $X_1$ more strongly than with $X_2$. However, as $cov(X_1,X_3)$ goes to ~0.921, the mean/median/mode of $X_3$ goes to 3!

I would expect that $X_3$ would be near $X_2$=1, or, at the very least, somewhere between $X_1$=-1, $X_2$=1 or the mean of 0. What is the intuition behind this 3?

  • 1
    $\begingroup$ Conditional distribution of $X_2 | X_1 = -1$ is N(-0.98,0.0396). So $X_2 = 1$ is 9.95 standard deviations out from its mean. So, rather unlikely. As a result, severe contortions can be necessary for distribution of $X_3|X_1 = -1,X_2 = 1$ Let $x = Cov(X_1,X_3)$. Then when x = 1 or its minimum value of 0.92708, the conditional variance of $X_3 = 0$, so the conditional distribution of $X_3$ boils down to conditional mean. For x =1, this is at $X_1 = -1$. For x = 0.98 ("neutral") ,halfway at 0. Another 0.02 down to 0.96 gets us to $X_2 = 1. And below that, extreme contortions are necessary. $\endgroup$ – Mark L. Stone Apr 13 '18 at 1:13
  • 1
    $\begingroup$ Conditional distribution of $X_3|X_1 = -1, X_2 = 1$ never has a very high variance. That conditional) variance is maximized at $x = 0.98^2$, with a value of 0.0396. So there's never much wiggle room to be consistent with specified 3D Normal. This may not be an intuitive explanation, but it's what I have. I upvoted the question (still the only upvote) because I thought that there doesn't seem to be an obvious "intuitive" answer. I waited to post these comments to see if someone would post a good answer which satisfied me. $\endgroup$ – Mark L. Stone Apr 13 '18 at 1:20
  • $\begingroup$ Yeah, I had noticed that $p(X_3\&X_1=-1\&X_2=1)$ was insanely small, and had wondered if this was related as well. $\endgroup$ – Scott Apr 13 '18 at 2:58
  • $\begingroup$ Incidentally, this problem is arising out of Gaussian Process Regression, where this interesting property of multivariate gaussians is used to capture trends in time series. $\endgroup$ – Scott Apr 13 '18 at 2:58
  • $\begingroup$ For the record, typo in my 1st comment: min value of x = 0.9208. $\endgroup$ – Mark L. Stone Apr 13 '18 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.