1
$\begingroup$

I wonder how I can calculate the coeffecients of a multiple linear regression, given just the mean and covariance matrix.

For example with this values:

Model: $Y = \beta_0 + \beta_1 \cdot X_1 + \beta_2 \cdot X_2$

$\bar{Y} = 2, \bar{X_1} = 3, \bar{X_2} = 6$ and the covarianze matrix like $\left( \begin{array}{rrr} 3 & 1 & 1 \\ 1 & 5 & -9 \\ 1 & -9 & 25 \\ \end{array}\right) $

Can anybody tell me how to get the values for $\beta_i$?

$\endgroup$

2 Answers 2

3
$\begingroup$

Make use of the standard covariance properties:

$Cov(aX, bY) = a\cdot b\cdot Cov(X, Y)\\ Cov(X+c, Y) = Cov(X, Y)\\ Cov(X+Y, Z) = Cov(X, Z) + Cov(Y, Z)\\ Cov(X, X) = Var(X)$

where $a$, $b$ and $c$ are constant values, and $X$, $Y$ and $Z$ are random variables.

With your situation, you can write for instance $Cov(Y, X)$ and make $\beta_1$ and $\beta_2$ appear by replacing $Y$ by its linear regression expression.

$\endgroup$
4
  • $\begingroup$ And what linear regression expression do you mean? $\endgroup$
    – T. Beige
    Sep 28, 2018 at 8:55
  • 1
    $\begingroup$ $Cov(Y, X_1) = Cov(\beta_0 + \beta_1\cdot X_1 + \beta_2\cdot X_2, X_1) = \beta_1 Var(X_1) + \beta_2 Cov(X_2, X_1)$. Since $Var(X_1)$ and $Cov(X_1, X_2)$ are known from your covariance matrix, this gives you a first equation for $\beta_1$ and $\beta_2$. Repeat this for $Cov(Y, X2)$, $Var(Y)$, and so on, and you get enough equations to solve the system and get $\beta_1$ and $\beta_2$. $\endgroup$ Sep 28, 2018 at 10:17
  • $\begingroup$ Ah, I understand your way of doing this. But I won't get $\beta_0$ with covariances. Or is it correct to take $\bar{Y} = \beta_0 + \beta_1 \bar{X}_1+ \beta_2 \bar{X}_2 $? $\endgroup$
    – T. Beige
    Sep 28, 2018 at 11:50
  • 3
    $\begingroup$ For linear regression it holds that: $\overline{y}=\beta_0 + \beta_1 \overline{x_1}+ \beta_2 \overline{x_2}$, therefore if you calculated all betas but the intercept you can simply solve the above equatino for $\beta_0$ $\endgroup$
    – Sebastian
    Sep 28, 2018 at 12:16
1
$\begingroup$

I give you an answer to calculate the coefficients using the inverse of the Covariance Matrix, which is also referred to as the Anti-Image Covariance Matrix

In simple linear regression: $Y=\beta_0+\beta_1X$ you can write $\beta_1=\frac{cov(x,y)}{var(x)}$ and then you easily obtain $\beta_0$ as $\overline{y}=\beta_0+\beta_1\overline{x}$

Now the problem if you have more than one predictor Variable as e.g. in your example: $Y=\beta_0 +\beta_1 X_2+\beta_2 X_2$ is that you can also have covariance between $X_1$ and $X_2$. So it is not anymore possible to simply set $\beta_1=\frac{cov(y,x_1)}{var(x_1)}$ and $\beta_2=\frac{cov(y,x_2)}{var(x_2)}$ due to the problem of Collinearity.

So what one would like to do is remove the linear influence of $X_2$ on $X_1$ and the other way around. So it turns out that we can calculate $\beta_1=\frac{cov(y,r_1)}{var(r_1)}$, where $r_1$ is the vector of residuals of the linear regression $Y=\beta X_1$, and analogously $\beta_2=\frac{cov(y,r_2)}{var(r_2)}$, where $r_2$ is the vector of residuals ofthe linear regression $Y=\beta X_2$. As I said, you can imagine this of removing the linear part of $X_1$ that is due to $X_2$ and the other way around.

I am now going to explain how this is connected with the Inverse of the Covariance Matrix, I will explain it in the 3d case because of notation but it works equally well for larger $n$. (This notation of using $X_1,X_2$now has nothing to do with the notation of your example)

Assume that you have a matrix $(X_1,X_2,X_3)$ with $X_i \in \mathbb{R}^k$. We denote with $C$ the covariance matrix of these 3 vectors, and with $\tilde{c_{i,j}}$ the elements of the inverse of $C$. Now it turns out that $\tilde{c}_{i,i}= \frac{1}{var(\tilde{x}_i)}$, where $\tilde{x_i}$ is the residual vector of the linear regression $X_i=\beta_0 X_j+\beta_1 X_l$, where $i\neq l\neq j$ and $i,j,l \in \{1,2,3\}$, i.e. the diagonal elements are the inverse of the so called partial variances.

Furthermore $\tilde{c}_{i,j}=-\frac{cor(\tilde{x}_i,\tilde{x}_j)}{var(\tilde{x_i})var(\tilde{x_j})}$.

Note that the role of $\tilde{x_i}$ differs from $r_i$ above, as $\tilde{x_i}$ is the residual when one uses ALL other Collumns in the Matrix in the regression, while above for $r_i$ we only used the Collumns of the other PREDICTORS.

However due to some nice mathematical equalities it will still work out, as it holds that $$\frac{cov(y,r_1)}{var(r_1)}=\frac{cor(y,r_1)\sqrt{var(y)}\sqrt{var(r_1}}{var(r_1)}= \frac{cor(y,r_1)\sqrt{var(y}}{\sqrt{var(r_1)}})=\frac{-cov(\tilde{y},\tilde{x_1})}{var(\tilde{x_1})}= \quad -w_{1,2}\cdot\frac{1}{w_{2,2}}$$ when we denote with $\tilde{y} $ the residuals of predicting $Y=\beta_0 X_1 + \beta_1 X_2$ and with $\tilde{x_1}$ the residuals of $X_1 = \beta_0 Y + \beta_1 X_2$ and if we denote with $w_{i,j}$ the elements of the Inverse of the Covariance Matrix of $(Y,X_1,X_2)$ from your example. This would for example mean that $w_{1,1}=\frac{1}{var(\tilde{y})}$, or $w_{1,2}=\frac{-cor(\tilde{y},\tilde{x_1})}{\sqrt{var(\tilde{y})}\sqrt{var(\tilde{x_1})}}$.

$\endgroup$
2
  • $\begingroup$ Okay, but $ \omega_{1,1} = \frac{1}{var(\bar{y})}$ should be $ \omega_{1,1} = \frac{1}{var(y)}$ or is $Var(y) = Var(\bar{y})$? Or is $Var(y) = n \cdot Var(\bar{y})$ $\endgroup$
    – T. Beige
    Sep 28, 2018 at 11:43
  • 1
    $\begingroup$ it is not the mean of $y$ but a $y$ with a ~ above. $w_{1,1}$ is the invese of the partial variance of y. The partial variance of $y$ is the variance of the residual vector of the liner regression $Y=\beta_0 X_1 + \beta_1 X_2$, i.e. $w_{1,1}=\frac{1}{var(y-\hat{y})}$ $\endgroup$
    – Sebastian
    Sep 28, 2018 at 12:15

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .