2
$\begingroup$

The text I'm reading claims that for multivariate $X$, $Y$, and $Z$, and $\hat X$ and $\hat Y$ are the regression of $X$ and $Y$ on $Z$ we have that

$$\operatorname{Corr}\left(X, Y\middle|Z\right) = \operatorname{Corr}(X-\hat X, Y-\hat Y)$$

However, there was no derivation of this claim. What steps are required to show this identity, and why is the assumption of normal variables required?

$\endgroup$
  • $\begingroup$ Do you mean $Corr(X|Z, Y|Z)$, or just $Corr(X, Y|Z)$? In the first, you regress Z out of both X and Y; this is the partial correlation between X and Y conditional on Z. In the second, you regress Z out of only Y; this is a semipartial correlation with Y conditional on Z. $\endgroup$ – Noah Jun 25 '18 at 20:45
  • $\begingroup$ Ah, that's unfortunate notation. I intended $\operatorname{Corr}\left((X,Y)\middle|Z\right)$, which I believe corresponds to $\operatorname{Corr}\left(X|Z,Y|Z\right)$? $\endgroup$ – Frank Vel Jun 25 '18 at 20:57
  • $\begingroup$ $\operatorname{Corr}(X,Y|Z)$ is perfectly fine notation, see e.g. en.wikipedia.org/wiki/Law_of_total_covariance whereas $\operatorname{Corr}(X|Z,Y|Z)$ is not. $\endgroup$ – Jarle Tufto Jun 27 '18 at 11:31
1
$\begingroup$

This is definitional and comes from the fact that $$X_i = Z_i\beta_x + \epsilon_{xi}$$, $$\hat{X}_i = Z_i\beta_x$$ and $$X_i|Z_i = \epsilon_{xi} = X_i - Z_i\beta_x = X_i - \hat{X}_i$$ (and the same for $Y$). These all come from the definition of regression. Then you can simply substitute: $$Corr(X,Y|Z) = Corr(X|Z, Y|Z) = Corr(X - \hat{X}, Y - \hat{Y})$$.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ So there is no need to assume the variables to be normal? $\endgroup$ – Frank Vel Jun 25 '18 at 18:59
  • $\begingroup$ Perhaps I'm wrong (I think I received a downvote for this one), but I don't think you need that assumption. $\endgroup$ – Noah Jun 25 '18 at 20:43
  • 2
    $\begingroup$ This doesn't look right to me. Your equation $\hat{X}_i = Z_i \beta_x$ uses the true coefficient $\beta_x$ rather than the corresponding OLS estimator. You seem to be looking at the true error terms rather than residuals. $\endgroup$ – Ben Jun 26 '18 at 3:11
  • $\begingroup$ That's true, and probably an oversimplification, but I believe this identity works on both the error term and the residuals, i.e., in both the population (using the true $\beta_x$) and the sample (using the estimated $\hat{\beta}_x$). The quantity $X|Z$ doesn't imply that we're in sample mode, and nothing about the structure of the proof necessitates examining either the sample or population. $\endgroup$ – Noah Jun 26 '18 at 13:58
  • 1
    $\begingroup$ @MartijnWeterings In general $\operatorname{Corr}(X,Y|Z)$ depends on the value $Z$ we condition on. So the left hand side don't necessarily equal the right hand side (which is just some constant) except in the multvariate normal case. $\endgroup$ – Jarle Tufto Jun 27 '18 at 13:00
1
$\begingroup$

For a multivariate normal random vector partitioned as $\mathbf{x}=\left[\begin{matrix}\mathbf{x}_1 \\ \mathbf{x}_2\end{matrix}\right]$, the conditional distribution of $\mathbf{x}_1$ conditional on on $\mathbf{x}_2$ is also multivariate normal with mean vector equal to the general expression for the best linear minimum mean square error predictor $\hat{\mathbf{x}}_1$ of $\mathbf{x}_1$ based on $\mathbf{x}_2$ (applying to any multivariate distribution) and, importantly, a variance-covariance matrix that does not depend on $\mathbf{x}_2$.

In the multivariate normal case we can thus write $$ \mathbf{x}_1=\hat{\mathbf{x}}_1 + \mathbf{e}_1, $$ where $\mathbf{e}_1$ is independent of $\hat{\mathbf{x}}_1$ and multivariate normal with zero mean and variance-covariance matrix equal to the the above conditional variance-covariance matrix.

Hence, $$ \operatorname{Var}(\mathbf{x}_1-\hat{\mathbf{x}}_1) = \operatorname{Var}\mathbf{e}_1=\operatorname{Var}(\mathbf{x}_1|\mathbf{x}_2), $$ that is, all variances, covariances and correlations of deviations from $\hat{\mathbf{x}}_1$ equals the corresponding conditional quantities.

This equality does not hold in general, however, as we can easily construct a multivariate distribution for which the conditional variance-covariance matrix of $\mathbf{x}_1$ depends on $\mathbf{x}_2$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I believe I am a bit stuck in definitions. I get that the $x_2$ dependency of $Var(x_1|x_2)$ but would not equally the distribution of the heterogeneous error-term $Var(x_1-\hat{x}_1)$ change depending on $x_2$? Or is this term related to the residuals? Or is the term not considered to be conditional or evaluated over all $x_2$? It is a bit confusing that $\hat{x}_1$ is ambiguous. $\endgroup$ – Sextus Empiricus Jun 27 '18 at 15:40
  • $\begingroup$ $\hat x_1=\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2)$ is just another random variable defined on the same sample space as $x_1$ so $\operatorname{Var}(x_1-\hat x_1)$ is just some constant whereas $\operatorname{Var}(x_1|x_2)$ may depend on $x_2$. $\endgroup$ – Jarle Tufto Jun 27 '18 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.