Multiple correlation formula

Question

Given $v \sim N_{p+1}(\mu, \Sigma), v = (x, y)', \mu = (\mu_x, \mu_y)', \Sigma = \begin{pmatrix} \Sigma_{xx} & \sigma_{xy} \\ \sigma_{yx} & \sigma_{yy} \end{pmatrix}$, how do you prove the formula for multiple correlation (the Pearson correlation between $\mu_{y|x}$ and $y$): $$\rho(y,x) = Corr(y, E[y|x]) = \left( \frac{\sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy}}{\sigma_{yy}} \right)^{1/2}$$ I tried to use $\mu_{y|x} = \mu_y - \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x)$ and plug it into $Corr(y, E[y|x]) = \frac{ Cov(y, E[y|x])}{\sqrt{V(y)}\sqrt{V(E[y|x])}}$ without any success.

Answer 1

The initial approach can be used as well:

$$\rho(x, y) = \frac{\text{Cov}(\mu_{y|x}, y)}{\sqrt{V(\mu_{y|x}) V(y)}}.$$

Then, $$V(y) = \sigma_{yy},$$

\begin{align*} V(\mu_{y|x}) &= V(\mu_y + \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x)) \\ &= V(\sigma_{yx}\Sigma_{xx}^{-1}x) && \small{ | \mu_y \text{ and } \sigma_{yx}\Sigma_{xx}^{-1}\mu_x \text{ are constants} }\\ &= \sigma_{yx}\Sigma_{xx}^{-1}V(x)(\sigma_{yx}\Sigma_{xx}^{-1})' && \small{ | \sigma_{xy} = \sigma_{yx}' \text{ and } \Sigma_{xx}^{-1} = (\Sigma_{xx}^{-1})' } \\ &= \sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xx}\Sigma_{xx}^{-1}\sigma_{xy} \\ &= \sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy} \end{align*} and

\begin{align*} \text{Cov}(\mu_{y|x}, y) &= \text{Cov}(\mu_{y} + \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x), y) \\ &= \text{Cov}(\sigma_{yx}\Sigma_{xx}^{-1}x, y) \\ &= \sigma_{yx}\Sigma_{xx}^{-1} \text{Cov}(x, y) \\ &= \sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy} \end{align*} Here, we used $\text{Cov}(ax + b, cy) = a\text{Cov}(x, y)c'$ and $V(x) = \text{Cov}(x,x)$. We can plug these results back into the formula and simplify:

\begin{align} \rho(x, y) &= \frac{\text{Cov}(\mu_{y|x}, y)}{\sqrt{V(\mu_{y|x}) V(y)}} \\ &= \frac{\sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy}}{\sqrt{\sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy} \sigma_{yy}}} \\ &= \sqrt{\frac{\sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy}}{\sigma_{yy}}} \end{align}

Answer 2

For a general succinct elucidation, a different notation is resorted for the sake of convenience and clarity. I would be paraphrasing the derivation in [1].

---

Let the random vector $\mathbf X_{p\times 1}$ with its covariance matrix $\mathbf \Sigma_{p\times p}$ be partitioned as

$$ \mathbf X_{p\times 1} = \begin{bmatrix}X_1\\ \mathbf X_{2,~(p-1) \times 1}\end{bmatrix},\tag{1.I}$$

$$\mathbf \Sigma_{p\times p} = \begin{bmatrix}\sigma_{11} & \boldsymbol\sigma^\mathsf T_{12,~1\times (p-1) }\\ \boldsymbol\sigma_{12,~(p-1)\times 1}& \mathbf \Sigma_{22,~(p-1)\times (p-1) }\end{bmatrix},\tag{1.II}$$

Multiple correlation coefficient between $X_1$ and $\mathbf X_2 ~(\bar R_{1\cdot2,\cdots, p})$ is the maximum correlation between $X_1$ and any linear function of $\mathbf X_2,$ that is, $\boldsymbol \alpha^\mathsf T\mathbf X_2.$

One handy approach of maximizing anything is to involve Cauchy-Schwarz Inequality (When does the equality occur? To be noted).

Now, by definition

\begin{align}\bar R_{1\cdot2,\cdots, p} &= \sup_{\boldsymbol\alpha} \frac{\operatorname{Cov}\left(X_1,\boldsymbol \alpha^\mathsf T\mathbf X_2\right)}{\left[\operatorname{Var}(X_1)\operatorname{Var}\left( \boldsymbol \alpha^\mathsf T\mathbf X_2\right) \right]^{\frac12}}\tag 2\\ & = \sup_{\boldsymbol\alpha} \frac{\boldsymbol\alpha^\mathsf T\boldsymbol\sigma_{12}}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12}\\ &=\sup_{\boldsymbol\alpha} \frac{\underbrace{\boldsymbol\alpha^\mathsf T\boldsymbol \Sigma_{22}^{1/2}}_{:=\mathbf u^\mathsf T}\underbrace{\boldsymbol \Sigma_{22}^{-1/2}\boldsymbol\sigma_{12}}_{:=\mathbf v}}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12} \\ &=\sup_{\boldsymbol\alpha} \frac{\mathbf u^\mathsf T\mathbf v}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12}\tag{2.I}\end{align}

By CS Inequality, $\mathbf u^\mathsf T\mathbf v \leqslant \Vert \mathbf u\Vert\Vert \mathbf v\Vert;$ from $\rm (2.I) ,$ it yields

\begin{align}\bar R_{1\cdot2,\cdots, p} &= \frac{\left(\mathbf u^\mathsf T\mathbf u\right)^\frac12\left(\mathbf v^\mathsf T \mathbf v\right)^\frac12 }{(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha)^\frac12}\\ &=\frac{\left( \boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12\left(\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12}\right)^\frac12}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12} \\ &= \left(\frac{\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12}}{\sigma_{11}}\right)^\frac12.\tag 3\end{align}

$(3) $ is the required result when $\boldsymbol \alpha =\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12} . $ Therefore, $\bar R_{1\cdot2,\cdots, p}$ is nothing but the correlation between $X_1$ and $\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\mathbf X_2.$

With additional imposition of multinormality on $\mathbf X, $ one can further refine $(3). $ One can even generalize $(3) $ when $$ \mathbf X_{p\times 1 }= \begin{bmatrix}\mathbf X_{1,~k\times 1}\\ \mathbf X_{2,~(p-k) \times 1}\end{bmatrix},\tag{4.I}$$

$$\mathbf \Sigma_{p\times p}= \begin{bmatrix}\boldsymbol\Sigma_{11,~k\times k} & \boldsymbol\Sigma_{12,~k\times (p-k) }\\ \boldsymbol\Sigma_{12,~(p-k)\times k}& \mathbf \Sigma_{22,~(p-k)\times (p-k) }\end{bmatrix}.\tag{4.II}$$ Then, if $X_i, ~i\in\{1, 2,\ldots, k\}$ be an element of the subvector $\mathbf X_1, $ multiple correlation coefficient between $X_i$ and the variables $X_{k+1}, \ldots, X_p$ in $\mathbf X_2,~(\bar R_{i\cdot k+1,\cdots ,p})$ is the maximum correlation coefficient between $X_i$ and any $\boldsymbol\alpha^\mathsf T\mathbf X_2$ which analogously would be $$\bar R_{i\cdot k+1,\cdots, p} = \left(\frac{\boldsymbol\sigma_{i}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{i}}{\boldsymbol\sigma_{ii}}\right)^\frac12, \tag 5$$ $\boldsymbol\sigma_{i}^\mathsf T$ being the $i$ th row of $\mathbf \Sigma_{12}$ and the maximizing $\boldsymbol\alpha = \boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{i}.$

---

Reference:

[1] Aspects of Multivariate Statistical Theory, Robb J. Muirhead, John Wiley & Sons, Inc., 1982.