1
$\begingroup$

I wonder how I can calculate the coeffecients of a multiple linear regression, given just the mean and covariance matrix.

For example with this values:

Model: $Y = \beta_0 + \beta_1 \cdot X_1 + \beta_2 \cdot X_2$

$\bar{Y} = 2, \bar{X_1} = 3, \bar{X_2} = 6$ and the covarianze matrix like $\left( \begin{array}{rrr} 3 & 1 & 1 \\ 1 & 5 & -9 \\ 1 & -9 & 25 \\ \end{array}\right) $

Can anybody tell me how to get the values for $\beta_i$?

$\endgroup$

2 Answers 2

3
$\begingroup$

Make use of the standard covariance properties:

$Cov(aX, bY) = a\cdot b\cdot Cov(X, Y)\\ Cov(X+c, Y) = Cov(X, Y)\\ Cov(X+Y, Z) = Cov(X, Z) + Cov(Y, Z)\\ Cov(X, X) = Var(X)$

where $a$, $b$ and $c$ are constant values, and $X$, $Y$ and $Z$ are random variables.

With your situation, you can write for instance $Cov(Y, X)$ and make $\beta_1$ and $\beta_2$ appear by replacing $Y$ by its linear regression expression.

$\endgroup$
4
  • $\begingroup$ And what linear regression expression do you mean? $\endgroup$
    – T. Beige
    Sep 28, 2018 at 8:55
  • 1
    $\begingroup$ $Cov(Y, X_1) = Cov(\beta_0 + \beta_1\cdot X_1 + \beta_2\cdot X_2, X_1) = \beta_1 Var(X_1) + \beta_2 Cov(X_2, X_1)$. Since $Var(X_1)$ and $Cov(X_1, X_2)$ are known from your covariance matrix, this gives you a first equation for $\beta_1$ and $\beta_2$. Repeat this for $Cov(Y, X2)$, $Var(Y)$, and so on, and you get enough equations to solve the system and get $\beta_1$ and $\beta_2$. $\endgroup$ Sep 28, 2018 at 10:17
  • $\begingroup$ Ah, I understand your way of doing this. But I won't get $\beta_0$ with covariances. Or is it correct to take $\bar{Y} = \beta_0 + \beta_1 \bar{X}_1+ \beta_2 \bar{X}_2 $? $\endgroup$
    – T. Beige
    Sep 28, 2018 at 11:50
  • 3
    $\begingroup$ For linear regression it holds that: $\overline{y}=\beta_0 + \beta_1 \overline{x_1}+ \beta_2 \overline{x_2}$, therefore if you calculated all betas but the intercept you can simply solve the above equatino for $\beta_0$ $\endgroup$
    – Sebastian
    Sep 28, 2018 at 12:16
1
$\begingroup$

I give you an answer to calculate the coefficients using the inverse of the Covariance Matrix, which is also referred to as the Anti-Image Covariance Matrix

In simple linear regression: $Y=\beta_0+\beta_1X$ you can write $\beta_1=\frac{cov(x,y)}{var(x)}$ and then you easily obtain $\beta_0$ as $\overline{y}=\beta_0+\beta_1\overline{x}$

Now the problem if you have more than one predictor Variable as e.g. in your example: $Y=\beta_0 +\beta_1 X_2+\beta_2 X_2$ is that you can also have covariance between $X_1$ and $X_2$. So it is not anymore possible to simply set $\beta_1=\frac{cov(y,x_1)}{var(x_1)}$ and $\beta_2=\frac{cov(y,x_2)}{var(x_2)}$ due to the problem of Collinearity.

So what one would like to do is remove the linear influence of $X_2$ on $X_1$ and the other way around. So it turns out that we can calculate $\beta_1=\frac{cov(y,r_1)}{var(r_1)}$, where $r_1$ is the vector of residuals of the linear regression $Y=\beta X_1$, and analogously $\beta_2=\frac{cov(y,r_2)}{var(r_2)}$, where $r_2$ is the vector of residuals ofthe linear regression $Y=\beta X_2$. As I said, you can imagine this of removing the linear part of $X_1$ that is due to $X_2$ and the other way around.

I am now going to explain how this is connected with the Inverse of the Covariance Matrix, I will explain it in the 3d case because of notation but it works equally well for larger $n$. (This notation of using $X_1,X_2$now has nothing to do with the notation of your example)

Assume that you have a matrix $(X_1,X_2,X_3)$ with $X_i \in \mathbb{R}^k$. We denote with $C$ the covariance matrix of these 3 vectors, and with $\tilde{c_{i,j}}$ the elements of the inverse of $C$. Now it turns out that $\tilde{c}_{i,i}= \frac{1}{var(\tilde{x}_i)}$, where $\tilde{x_i}$ is the residual vector of the linear regression $X_i=\beta_0 X_j+\beta_1 X_l$, where $i\neq l\neq j$ and $i,j,l \in \{1,2,3\}$, i.e. the diagonal elements are the inverse of the so called partial variances.

Furthermore $\tilde{c}_{i,j}=-\frac{cor(\tilde{x}_i,\tilde{x}_j)}{var(\tilde{x_i})var(\tilde{x_j})}$.

Note that the role of $\tilde{x_i}$ differs from $r_i$ above, as $\tilde{x_i}$ is the residual when one uses ALL other Collumns in the Matrix in the regression, while above for $r_i$ we only used the Collumns of the other PREDICTORS.

However due to some nice mathematical equalities it will still work out, as it holds that $$\frac{cov(y,r_1)}{var(r_1)}=\frac{cor(y,r_1)\sqrt{var(y)}\sqrt{var(r_1}}{var(r_1)}= \frac{cor(y,r_1)\sqrt{var(y}}{\sqrt{var(r_1)}})=\frac{-cov(\tilde{y},\tilde{x_1})}{var(\tilde{x_1})}= \quad -w_{1,2}\cdot\frac{1}{w_{2,2}}$$ when we denote with $\tilde{y} $ the residuals of predicting $Y=\beta_0 X_1 + \beta_1 X_2$ and with $\tilde{x_1}$ the residuals of $X_1 = \beta_0 Y + \beta_1 X_2$ and if we denote with $w_{i,j}$ the elements of the Inverse of the Covariance Matrix of $(Y,X_1,X_2)$ from your example. This would for example mean that $w_{1,1}=\frac{1}{var(\tilde{y})}$, or $w_{1,2}=\frac{-cor(\tilde{y},\tilde{x_1})}{\sqrt{var(\tilde{y})}\sqrt{var(\tilde{x_1})}}$.

$\endgroup$
2
  • $\begingroup$ Okay, but $ \omega_{1,1} = \frac{1}{var(\bar{y})}$ should be $ \omega_{1,1} = \frac{1}{var(y)}$ or is $Var(y) = Var(\bar{y})$? Or is $Var(y) = n \cdot Var(\bar{y})$ $\endgroup$
    – T. Beige
    Sep 28, 2018 at 11:43
  • 1
    $\begingroup$ it is not the mean of $y$ but a $y$ with a ~ above. $w_{1,1}$ is the invese of the partial variance of y. The partial variance of $y$ is the variance of the residual vector of the liner regression $Y=\beta_0 X_1 + \beta_1 X_2$, i.e. $w_{1,1}=\frac{1}{var(y-\hat{y})}$ $\endgroup$
    – Sebastian
    Sep 28, 2018 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.