# Multiple correlation formula

Given $$v \sim N_{p+1}(\mu, \Sigma), v = (x, y)', \mu = (\mu_x, \mu_y)', \Sigma = \begin{pmatrix} \Sigma_{xx} & \sigma_{xy} \\ \sigma_{yx} & \sigma_{yy} \end{pmatrix}$$, how do you prove the formula for multiple correlation (the Pearson correlation between $$\mu_{y|x}$$ and $$y$$): $$\rho(y,x) = Corr(y, E[y|x]) = \left( \frac{\sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy}}{\sigma_{yy}} \right)^{1/2}$$ I tried to use $$\mu_{y|x} = \mu_y - \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x)$$ and plug it into $$Corr(y, E[y|x]) = \frac{ Cov(y, E[y|x])}{\sqrt{V(y)}\sqrt{V(E[y|x])}}$$ without any success.

The initial approach can be used as well:

$$\rho(x, y) = \frac{\text{Cov}(\mu_{y|x}, y)}{\sqrt{V(\mu_{y|x}) V(y)}}.$$

Then, $$V(y) = \sigma_{yy},$$

\begin{align*} V(\mu_{y|x}) &= V(\mu_y + \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x)) \\ &= V(\sigma_{yx}\Sigma_{xx}^{-1}x) && \small{ | \mu_y \text{ and } \sigma_{yx}\Sigma_{xx}^{-1}\mu_x \text{ are constants} }\\ &= \sigma_{yx}\Sigma_{xx}^{-1}V(x)(\sigma_{yx}\Sigma_{xx}^{-1})' && \small{ | \sigma_{xy} = \sigma_{yx}' \text{ and } \Sigma_{xx}^{-1} = (\Sigma_{xx}^{-1})' } \\ &= \sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xx}\Sigma_{xx}^{-1}\sigma_{xy} \\ &= \sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy} \end{align*} and

\begin{align*} \text{Cov}(\mu_{y|x}, y) &= \text{Cov}(\mu_{y} + \sigma_{yx}\Sigma_{xx}^{-1}(x-\mu_x), y) \\ &= \text{Cov}(\sigma_{yx}\Sigma_{xx}^{-1}x, y) \\ &= \sigma_{yx}\Sigma_{xx}^{-1} \text{Cov}(x, y) \\ &= \sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy} \end{align*} Here, we used $$\text{Cov}(ax + b, cy) = a\text{Cov}(x, y)c'$$ and $$V(x) = \text{Cov}(x,x)$$. We can plug these results back into the formula and simplify:

\begin{align} \rho(x, y) &= \frac{\text{Cov}(\mu_{y|x}, y)}{\sqrt{V(\mu_{y|x}) V(y)}} \\ &= \frac{\sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy}}{\sqrt{\sigma_{yx}\Sigma_{xx}^{-1}\sigma_{xy} \sigma_{yy}}} \\ &= \sqrt{\frac{\sigma_{yx}\Sigma_{xx}^{-1} \sigma_{xy}}{\sigma_{yy}}} \end{align}

For a general succinct elucidation, a different notation is resorted for the sake of convenience and clarity. I would be paraphrasing the derivation in [1].

Let the random vector $$\mathbf X_{p\times 1}$$ with its covariance matrix $$\mathbf \Sigma_{p\times p}$$ be partitioned as

$$\mathbf X_{p\times 1} = \begin{bmatrix}X_1\\ \mathbf X_{2,~(p-1) \times 1}\end{bmatrix},\tag{1.I}$$

$$\mathbf \Sigma_{p\times p} = \begin{bmatrix}\sigma_{11} & \boldsymbol\sigma^\mathsf T_{12,~1\times (p-1) }\\ \boldsymbol\sigma_{12,~(p-1)\times 1}& \mathbf \Sigma_{22,~(p-1)\times (p-1) }\end{bmatrix},\tag{1.II}$$

Multiple correlation coefficient between $$X_1$$ and $$\mathbf X_2 ~(\bar R_{1\cdot2,\cdots, p})$$ is the maximum correlation between $$X_1$$ and any linear function of $$\mathbf X_2,$$ that is, $$\boldsymbol \alpha^\mathsf T\mathbf X_2.$$

One handy approach of maximizing anything is to involve Cauchy-Schwarz Inequality (When does the equality occur? To be noted).

Now, by definition

\begin{align}\bar R_{1\cdot2,\cdots, p} &= \sup_{\boldsymbol\alpha} \frac{\operatorname{Cov}\left(X_1,\boldsymbol \alpha^\mathsf T\mathbf X_2\right)}{\left[\operatorname{Var}(X_1)\operatorname{Var}\left( \boldsymbol \alpha^\mathsf T\mathbf X_2\right) \right]^{\frac12}}\tag 2\\ & = \sup_{\boldsymbol\alpha} \frac{\boldsymbol\alpha^\mathsf T\boldsymbol\sigma_{12}}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12}\\ &=\sup_{\boldsymbol\alpha} \frac{\underbrace{\boldsymbol\alpha^\mathsf T\boldsymbol \Sigma_{22}^{1/2}}_{:=\mathbf u^\mathsf T}\underbrace{\boldsymbol \Sigma_{22}^{-1/2}\boldsymbol\sigma_{12}}_{:=\mathbf v}}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12} \\ &=\sup_{\boldsymbol\alpha} \frac{\mathbf u^\mathsf T\mathbf v}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12}\tag{2.I}\end{align}

By CS Inequality, $$\mathbf u^\mathsf T\mathbf v \leqslant \Vert \mathbf u\Vert\Vert \mathbf v\Vert;$$ from $$\rm (2.I) ,$$ it yields

\begin{align}\bar R_{1\cdot2,\cdots, p} &= \frac{\left(\mathbf u^\mathsf T\mathbf u\right)^\frac12\left(\mathbf v^\mathsf T \mathbf v\right)^\frac12 }{(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha)^\frac12}\\ &=\frac{\left( \boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12\left(\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12}\right)^\frac12}{\left(\sigma_{11}\boldsymbol\alpha^\mathsf T\mathbf \Sigma_{22}\boldsymbol\alpha\right)^\frac12} \\ &= \left(\frac{\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12}}{\sigma_{11}}\right)^\frac12.\tag 3\end{align}

$$(3)$$ is the required result when $$\boldsymbol \alpha =\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{12} .$$ Therefore, $$\bar R_{1\cdot2,\cdots, p}$$ is nothing but the correlation between $$X_1$$ and $$\boldsymbol\sigma_{12}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\mathbf X_2.$$

With additional imposition of multinormality on $$\mathbf X,$$ one can further refine $$(3).$$ One can even generalize $$(3)$$ when $$\mathbf X_{p\times 1 }= \begin{bmatrix}\mathbf X_{1,~k\times 1}\\ \mathbf X_{2,~(p-k) \times 1}\end{bmatrix},\tag{4.I}$$

$$\mathbf \Sigma_{p\times p}= \begin{bmatrix}\boldsymbol\Sigma_{11,~k\times k} & \boldsymbol\Sigma_{12,~k\times (p-k) }\\ \boldsymbol\Sigma_{12,~(p-k)\times k}& \mathbf \Sigma_{22,~(p-k)\times (p-k) }\end{bmatrix}.\tag{4.II}$$ Then, if $$X_i, ~i\in\{1, 2,\ldots, k\}$$ be an element of the subvector $$\mathbf X_1,$$ multiple correlation coefficient between $$X_i$$ and the variables $$X_{k+1}, \ldots, X_p$$ in $$\mathbf X_2,~(\bar R_{i\cdot k+1,\cdots ,p})$$ is the maximum correlation coefficient between $$X_i$$ and any $$\boldsymbol\alpha^\mathsf T\mathbf X_2$$ which analogously would be $$\bar R_{i\cdot k+1,\cdots, p} = \left(\frac{\boldsymbol\sigma_{i}^\mathsf T\boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{i}}{\boldsymbol\sigma_{ii}}\right)^\frac12, \tag 5$$ $$\boldsymbol\sigma_{i}^\mathsf T$$ being the $$i$$ th row of $$\mathbf \Sigma_{12}$$ and the maximizing $$\boldsymbol\alpha = \boldsymbol\Sigma_{22}^{-1}\boldsymbol\sigma_{i}.$$

## Reference:

[1] Aspects of Multivariate Statistical Theory, Robb J. Muirhead, John Wiley & Sons, Inc., 1982.

• Thank you very much! Commented Sep 9, 2022 at 10:17