5
$\begingroup$

Given that $X$ and $Y$ are jointly normal, does it mean that $Z = Y - \rho\frac{\sigma_Y}{\sigma_X}X$ and $X$ are independent because the correlation between $Z$ and $X$ is zero?

I know that zero correlation implies independence for jointly normal RVs, but it is not clear to me that $Z$ and $X$ are jointly normal (btw, how do we know if a given transformation of normal RVs is jointly normal?) and it seem very counter-intuitive that they are independent since one is a function of the other.

$\endgroup$
5
$\begingroup$

Let the random vector $\mathbf{X} \sim N_{p}(\mathbf{\mu},\mathbf{\Sigma})$. If we partition $\mathbf{X}$ as $\left(\begin{array}{c} \mathbf{X^{(1)}}\\ \mathbf{X^{(2)}} \end{array}\right)$ and take a non-singular linear transformation to the components of $\mathbf{X}$ as \begin{eqnarray*} \mathbf{Y^{(1)}} &=& \mathbf{X^{(1)} + M X^{(2)}}\\ \mathbf{Y^{(2)}} &=& \mathbf{X^{(2)}} \end{eqnarray*} where the matrix $\mathbf{M}$ is chosen such that the sub-vectors $\mathbf{Y^{(1)}}$ and $\mathbf{Y^{(2)}}$ are uncorrelated. That is, Choose $\mathbf{M}$ such that, \begin{equation*} \mathbb{E}\mathbf{\left(Y^{(1)}-\mathbb{E}\mathbf{Y^{(1)}}\right)\left(Y^{(2)}-\mathbb{E}\mathbf{Y^{(2)}}\right)^{T}} = \mathbf{O} \end{equation*} Substituting $\mathbf{Y^{(1)}}$ and $\mathbf{Y^{(2)}}$ into the above equation and solving for $\mathbf{M},$ we get $\mathbf{M=-\Sigma_{12}\Sigma_{22}^{-1}}.$ In the bivariate case, let \begin{equation*} \mathbf{X} = \left(\begin{array}{c} \mathbf{X_{1}}\\ \mathbf{X_{2}} \end{array}\right) \end{equation*} \begin{equation*} \mathbf{\Sigma}= \left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \\ \end{array}\right) = \left(\begin{array}{cc} \sigma_{1}^{2} & \sigma_{12}\\ \sigma_{21} & \sigma_{2}^{2}\\ \end{array}\right) = \left(\begin{array}{cc} \sigma_{1}^{2} & \rho\sigma_{1}\sigma_{2}\\ \rho\sigma_{1}\sigma_{2} & \sigma_{2}^{2}\\ \end{array}\right) \end{equation*} From which we note that, $\Sigma_{12}=\rho\sigma_{1}\sigma_{2}$, $\Sigma_{22}^{-1}=\dfrac{1}{\sigma_{2}^{2}}$ and $-\mathbf{\Sigma_{12}\Sigma_{22}^{-1}} = -\rho\dfrac{\sigma_{1}}{\sigma_{2}} $. Hence, the random variables defined by

\begin{eqnarray*} Y_{1} &=& X_{1} - \mathbf{\Sigma_{12}\Sigma_{22}^{-1}}X_{2} = X_{1}-\rho\dfrac{\sigma_{1}}{\sigma_{2}}X_{2}\\ Y_{2} &=& X_{2}. \end{eqnarray*} are independent. Since, \begin{equation*} \mathbf{Y} = \left(\begin{array}{c} \mathbf{Y^{(1)}}\\ \mathbf{Y^{(2)}} \end{array}\right) = \left(\begin{array}{ll} \mathbf{I_{11}} & -\mathbf{\Sigma_{12}\Sigma_{22}^{-1}}\\ \mathbf{O} & \mathbf{I_{22}} \end{array}\right)\mathbf{X} \end{equation*} being a non-singular transformation of $\mathbf{X}$, the random vector $\mathbf{Y}$ has a Multivariate normal distribution with the variance-covariance matrix $\left(\begin{array}{ll} \mathbf{\Sigma_{11}}-\mathbf{\Sigma_{12}\Sigma_{22}^{-1}} & \mathbf{O}\\ \mathbf{O} & \mathbf{\Sigma_{22}} \end{array}\right).$

$\endgroup$
  • $\begingroup$ Thank you for your reply. I am not clear, however, on how you deduce that they are independent. As far as I can see you just showed that they are uncorrelated, but for this to imply independence, they would have to be jointly normal, which is not clear that they are (as I stated in the OP). $\endgroup$ – Confounded May 5 '18 at 20:40
  • $\begingroup$ @Confounded $\mathbf{X}$ is transformed to $\mathbf{Y}$ such that the sub-vectors of $\mathbf{Y}$ are independent. The sub-vectors of $\mathbf{Y}$ being linear transformations of $\mathbf{X}$ are in turn multivariate normal. The sub-vectors of $\mathbf{Y}$ are independent since their covariance matrix is $\mathbf{O}$. $\endgroup$ – L.V.Rao May 6 '18 at 1:25
  • $\begingroup$ @Confounded Gaussian random vectors have the property that if they are uncorrelated, they are independent. This is because when the cross terms of the covariance matrix is zero, the covariance matrix is block diagonal and hence the probability distribution factors into two parts, each being the pdfs of the subvectors $\endgroup$ – Arun Mar 27 at 12:09
3
$\begingroup$

Hint:

One can see that $Z$, being a linear combination of jointly normal variables $X$ and $Y$, is itself univariate normal. And two linear combinations (namely, $Z$ and $X$) of jointly normal variables are themselves jointly normal. So one possible way is to find the joint moment generating function of $(Z,X)$ to see whether $X$ and $Z$ are independent or not. The joint MGF of $(Z,X)$ is given by

$$M(t_1,t_2)=E(\exp(t_1Z+t_2X))=E\left[\exp\left(\left(t_2-\rho t_1\frac{\sigma_y}{\sigma_x}\right)X+t_1Y\right)\right]$$

From the expression of the joint MGF of $(X,Y)$, that last expectation gives $$M(t_1,t_2)=\exp\left[\frac{1}{2}\left(\sigma_x^2\left(t_2-t_1\rho\frac{\sigma_y}{\sigma_x}\right)^2+\sigma_y^2t_1^2+2\rho\sigma_x\sigma_y\left(t_2-t_1\rho\frac{\sigma_y}{\sigma_x}\right)t_1\right)\right]$$

Simplify that exponent in terms of a bivariate normal MGF and then try to conclude from the correlation whether $Z$ and $X$ are independent or not. You already know that zero correlation is a necessary and sufficient condition of independence for jointly normal variables.

$\endgroup$
  • $\begingroup$ Thanks for the hint. The MGF does indeed turn out to be in the form of independent bivariate normal. Still, very counter-intuitive that $Z$ and $X$ are independent even though $Z$ is a function of $X$. So, do any linear transformations of normals give multivariate normals? Are there any restriction on the transformation matrix A? If we start with $n$ normals X, and the matrix A is $m \times n$, does the transformation AX results in multivariate normals both when $m < n$ and $m>n$? Does A have to be full rank? $\endgroup$ – Confounded May 4 '18 at 15:06
  • $\begingroup$ @Confounded I don't know enough to answer those questions. But regarding what you say as counter-intuitive, I think it has got to do with the regression of $Y$ on $X$ being linear. $\endgroup$ – StubbornAtom May 4 '18 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.