7
$\begingroup$

Two variables that are uncorrelated are not necessarily independent, as is simply exemplified by the fact that $X$ and $X^2$ are uncorrelated but not independent. However, two variables that are uncorrelated AND jointly normally distributed are guaranteed to be independent. Can someone explain intuitively why this is true? What exactly does joint normality of two variables add to the knowledge of zero correlation between two variables, which leads us to conclude that these two variables MUST be independent?

$\endgroup$
  • 3
    $\begingroup$ It is not generally the case that $X$ and $X^2$ are uncorrelated (unless you put particular conditions on the $X$ that would make them uncorrelated, but you mention none). $\endgroup$ – Glen_b Nov 10 '18 at 0:03
  • $\begingroup$ First, going back to the fact that correlation refers to linear relationships, please explain how X^2 is linearly related to X. Second, you seem to be stating that not only can X^2 and X be linearly related, but that they are linearly related more often than not, given the use of the word "generally". Please explain. Thank you. $\endgroup$ – ColorStatistics Nov 10 '18 at 0:40
  • 1
    $\begingroup$ @Glen_b is spot on: $X$ and $X^{2}$ are only uncorrelated if you specifically stipulate the range of $X$. For example Pearson's $r \approx 0.98$ for $X$ and $X^{2}$ when restricting the sample of $X\sim \mathcal{N}(0,1)$ to values of $X$ in the range greater than 1. Check it out yourself (R): X <- rnorm(n=10000); X2 <- X*X; cor(X[X>1],X2[X>1]) $\endgroup$ – Alexis Nov 10 '18 at 0:51
  • 1
    $\begingroup$ @Alexis It's not just the range, but the how the probabilities distribute over those values within the range. If you change the distribution you change the correlation. $\endgroup$ – Glen_b Nov 10 '18 at 1:31
  • 3
    $\begingroup$ @ColorStatistics correlation is the degree of linear relationship, yes. However, the projection of $x^2$ onto $x$ may involve a substantial linear component. If you want to see an example with high linear correlation between a variable and its square, let $X$ take the values 0 and 1 with equal probability (e.g. record the number of heads in the toss of a single fair coin). Then corr$(X,X^2)=1$ (!). If you're free to specify the distribution of $X$, you can make the correlation between $X$ and $X^2$ take any value between $-1$ and $1$. ... ctd $\endgroup$ – Glen_b Nov 10 '18 at 1:41
6
$\begingroup$

The the joint probability density function (pdf) of bivariate normal distribution is: $$f(x_1,x_2)=\frac 1{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}}\exp[-\frac z{2(1-\rho^2)}], $$

where

$$z=\frac{(x_1-\mu_1)^2}{\sigma_1^2}-\frac{2\rho(x_1-\mu_1)(x_2-\mu_2)}{\sigma_1\sigma_2}+\frac{(x_2-\mu_2)^2}{\sigma_2^2}.$$ When $\rho = 0$, $$\begin{align}f(x_1,x_2) &=\frac 1{2\pi\sigma_1\sigma_2}\exp[-\frac 12\left\{\frac{(x_1-\mu_1)^2}{\sigma_1^2}+\frac{(x_2-\mu_2)^2}{\sigma_2^2}\right\} ]\\ & = \frac 1{\sqrt{2\pi}\sigma_1}\exp[-\frac 12\left\{\frac{(x_1-\mu_1)^2}{\sigma_1^2}\right\}] \frac 1{\sqrt{2\pi}\sigma_2}\exp[-\frac 12\left\{\frac{(x_2-\mu_2)^2}{\sigma_2^2}\right\}]\\ &= f(x_1)f(x_2)\end{align}$$.

So they are independent.

$\endgroup$
  • $\begingroup$ I was two lines slower than you! (+1) $\endgroup$ – jbowman Nov 9 '18 at 21:01
  • 1
    $\begingroup$ Thank you all. Elegant proof. It's clear now. It seems to me that given the flow of the proof, I should have asked what knowledge of zero correlation adds to knowledge of joint normality and not the other way around. $\endgroup$ – ColorStatistics Nov 9 '18 at 21:18
  • 1
    $\begingroup$ What about an intuitive explanation as to why it is true? $\endgroup$ – ColorStatistics Nov 10 '18 at 12:51
  • $\begingroup$ Maybe there is no intuitive simple explanation. $\endgroup$ – user158565 Nov 10 '18 at 15:34
  • $\begingroup$ Could we get some intuition along the line of reasoning that a Gaussian process' higher (than 2) moments are all zero, and adding the zero correlation condition (moment 2), pins down all moments greater than one to zero? $\endgroup$ – ColorStatistics Nov 11 '18 at 13:33
2
$\begingroup$

Joint normality of two random variables $X,Y$ can be characterized in either of two simple ways:

  • For every pair $a,b$ of (non-random) real numbers, $aX+bY$ has a univariate normal distribution.

  • There are random variables $Z_1,Z_2\sim\operatorname{\text{i.i.d.}} \operatorname N(0,1)$ and real numbers $a,b,c,d$ such that $$\begin{align} X & = aZ_1+bZ_2 \\ \text{and } Y & = cZ_1 + dZ_2. \end{align}$$

That the first of these follows from the second is easy to show. That the second follows from the first takes more work, and maybe I'll post on it soon . . .

If the second one it true, then $\operatorname{cov}(X,Y) = ac + bd.$

If this covariance is $0,$ then the vectors $(a,b),$ $(c,d)$ are orthogonal to each other. Then $X$ is a scalar multiple of the orthogonal projection of $(Z_1,Z_2)$ onto $(a,b)$ and $Y$ onto $(c,d).$

Now conjoin the fact of orthogonality with the circular symmetry of the joint density of $(Z_1,Z_2),$ to see that the distribution of $(X,Y)$ should be the same as the distribution of two random variables, one of which is a scalar multiple of the orthogonal projection of $(Z_1,Z_2)$ onto the $x$-axis, i.e. it is a scalar multiple of $Z_1,$ and the other is similarly a scalar multiple of $Z_2.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.