# Calculate coefficients in multiple linear regression with covariance matrix

I wonder how I can calculate the coeffecients of a multiple linear regression, given just the mean and covariance matrix.

For example with this values:

Model: $$Y = \beta_0 + \beta_1 \cdot X_1 + \beta_2 \cdot X_2$$

$$\bar{Y} = 2, \bar{X_1} = 3, \bar{X_2} = 6$$ and the covarianze matrix like $$\left( \begin{array}{rrr} 3 & 1 & 1 \\ 1 & 5 & -9 \\ 1 & -9 & 25 \\ \end{array}\right)$$

Can anybody tell me how to get the values for $$\beta_i$$?

Make use of the standard covariance properties:

$$Cov(aX, bY) = a\cdot b\cdot Cov(X, Y)\\ Cov(X+c, Y) = Cov(X, Y)\\ Cov(X+Y, Z) = Cov(X, Z) + Cov(Y, Z)\\ Cov(X, X) = Var(X)$$

where $$a$$, $$b$$ and $$c$$ are constant values, and $$X$$, $$Y$$ and $$Z$$ are random variables.

With your situation, you can write for instance $$Cov(Y, X)$$ and make $$\beta_1$$ and $$\beta_2$$ appear by replacing $$Y$$ by its linear regression expression.

• And what linear regression expression do you mean? Sep 28, 2018 at 8:55
• $Cov(Y, X_1) = Cov(\beta_0 + \beta_1\cdot X_1 + \beta_2\cdot X_2, X_1) = \beta_1 Var(X_1) + \beta_2 Cov(X_2, X_1)$. Since $Var(X_1)$ and $Cov(X_1, X_2)$ are known from your covariance matrix, this gives you a first equation for $\beta_1$ and $\beta_2$. Repeat this for $Cov(Y, X2)$, $Var(Y)$, and so on, and you get enough equations to solve the system and get $\beta_1$ and $\beta_2$. Sep 28, 2018 at 10:17
• Ah, I understand your way of doing this. But I won't get $\beta_0$ with covariances. Or is it correct to take $\bar{Y} = \beta_0 + \beta_1 \bar{X}_1+ \beta_2 \bar{X}_2$? Sep 28, 2018 at 11:50
• For linear regression it holds that: $\overline{y}=\beta_0 + \beta_1 \overline{x_1}+ \beta_2 \overline{x_2}$, therefore if you calculated all betas but the intercept you can simply solve the above equatino for $\beta_0$ Sep 28, 2018 at 12:16

I give you an answer to calculate the coefficients using the inverse of the Covariance Matrix, which is also referred to as the Anti-Image Covariance Matrix

In simple linear regression: $$Y=\beta_0+\beta_1X$$ you can write $$\beta_1=\frac{cov(x,y)}{var(x)}$$ and then you easily obtain $$\beta_0$$ as $$\overline{y}=\beta_0+\beta_1\overline{x}$$

Now the problem if you have more than one predictor Variable as e.g. in your example: $$Y=\beta_0 +\beta_1 X_2+\beta_2 X_2$$ is that you can also have covariance between $$X_1$$ and $$X_2$$. So it is not anymore possible to simply set $$\beta_1=\frac{cov(y,x_1)}{var(x_1)}$$ and $$\beta_2=\frac{cov(y,x_2)}{var(x_2)}$$ due to the problem of Collinearity.

So what one would like to do is remove the linear influence of $$X_2$$ on $$X_1$$ and the other way around. So it turns out that we can calculate $$\beta_1=\frac{cov(y,r_1)}{var(r_1)}$$, where $$r_1$$ is the vector of residuals of the linear regression $$Y=\beta X_1$$, and analogously $$\beta_2=\frac{cov(y,r_2)}{var(r_2)}$$, where $$r_2$$ is the vector of residuals ofthe linear regression $$Y=\beta X_2$$. As I said, you can imagine this of removing the linear part of $$X_1$$ that is due to $$X_2$$ and the other way around.

I am now going to explain how this is connected with the Inverse of the Covariance Matrix, I will explain it in the 3d case because of notation but it works equally well for larger $$n$$. (This notation of using $$X_1,X_2$$now has nothing to do with the notation of your example)

Assume that you have a matrix $$(X_1,X_2,X_3)$$ with $$X_i \in \mathbb{R}^k$$. We denote with $$C$$ the covariance matrix of these 3 vectors, and with $$\tilde{c_{i,j}}$$ the elements of the inverse of $$C$$. Now it turns out that $$\tilde{c}_{i,i}= \frac{1}{var(\tilde{x}_i)}$$, where $$\tilde{x_i}$$ is the residual vector of the linear regression $$X_i=\beta_0 X_j+\beta_1 X_l$$, where $$i\neq l\neq j$$ and $$i,j,l \in \{1,2,3\}$$, i.e. the diagonal elements are the inverse of the so called partial variances.

Furthermore $$\tilde{c}_{i,j}=-\frac{cor(\tilde{x}_i,\tilde{x}_j)}{var(\tilde{x_i})var(\tilde{x_j})}$$.

Note that the role of $$\tilde{x_i}$$ differs from $$r_i$$ above, as $$\tilde{x_i}$$ is the residual when one uses ALL other Collumns in the Matrix in the regression, while above for $$r_i$$ we only used the Collumns of the other PREDICTORS.

However due to some nice mathematical equalities it will still work out, as it holds that $$\frac{cov(y,r_1)}{var(r_1)}=\frac{cor(y,r_1)\sqrt{var(y)}\sqrt{var(r_1}}{var(r_1)}= \frac{cor(y,r_1)\sqrt{var(y}}{\sqrt{var(r_1)}})=\frac{-cov(\tilde{y},\tilde{x_1})}{var(\tilde{x_1})}= \quad -w_{1,2}\cdot\frac{1}{w_{2,2}}$$ when we denote with $$\tilde{y}$$ the residuals of predicting $$Y=\beta_0 X_1 + \beta_1 X_2$$ and with $$\tilde{x_1}$$ the residuals of $$X_1 = \beta_0 Y + \beta_1 X_2$$ and if we denote with $$w_{i,j}$$ the elements of the Inverse of the Covariance Matrix of $$(Y,X_1,X_2)$$ from your example. This would for example mean that $$w_{1,1}=\frac{1}{var(\tilde{y})}$$, or $$w_{1,2}=\frac{-cor(\tilde{y},\tilde{x_1})}{\sqrt{var(\tilde{y})}\sqrt{var(\tilde{x_1})}}$$.

• Okay, but $\omega_{1,1} = \frac{1}{var(\bar{y})}$ should be $\omega_{1,1} = \frac{1}{var(y)}$ or is $Var(y) = Var(\bar{y})$? Or is $Var(y) = n \cdot Var(\bar{y})$ Sep 28, 2018 at 11:43
• it is not the mean of $y$ but a $y$ with a ~ above. $w_{1,1}$ is the invese of the partial variance of y. The partial variance of $y$ is the variance of the residual vector of the liner regression $Y=\beta_0 X_1 + \beta_1 X_2$, i.e. $w_{1,1}=\frac{1}{var(y-\hat{y})}$ Sep 28, 2018 at 12:15