3
$\begingroup$

Let's say we have 2 predictors $X_1, X_2$. And there are $n=5$ samples. Say we have the following values for $X_1, X_2$:

$$ (1, -5) \\ (2, 0) \\ (3, 4) \\ (4, -3) \\ (5, 1) $$

$X_1$ and $X_2$ here are clearly orthogonal. If we include the intercept and introduce a column of ones, we find that for $y = [0,1,2,3,4]$

$$ \hat{\beta} = [-1, 1, 0] $$ which are the intercept, coefficient corresponding to $X_1, X_2$, respectively.

Now if I did simple linear regression for Y vs $X_1$ and $Y$ vs $X_2$. I find that $\hat{\beta}_1 = 1$, but $\hat{\beta}_2 = 0.18292683$. Why are they different here?

For orthogonal predictors shouldn't the single regression coefficients be the same as the multiple regression coefficients?

$\endgroup$

2 Answers 2

5
$\begingroup$

For orthogonal predictors shouldn't the single regression coefficients be the same as the multiple regression coefficients?

Yes, but only if you don't fit an intercept:

> X1 <- c(1,2,3,4,5)
> X2 <- c(-5,0,4,-3,1)
> 
> y <- c(0,1,2,3,4)
> 
> lm(y ~ -1 + X1 + X2) %>% summary()
Coefficients:
   Estimate Std. Error t value Pr(>|t|)   
X1  0.72727    0.06663   10.91  0.00165 **
X2  0.05882    0.06920    0.85  0.45774   

> 
> lm(y ~ -1 + X1) %>% summary()
Coefficients:
   Estimate Std. Error t value Pr(>|t|)    
X1  0.72727    0.06428   11.31 0.000348 ***

> 
> lm(y ~ -1 + X2) %>% summary()

Coefficients:
   Estimate Std. Error t value Pr(>|t|)
X2  0.05882    0.38235   0.154    0.885

$\endgroup$
8
  • $\begingroup$ If you center the predictors and response beforehand, then you are no longer fitting an intercept right? $\endgroup$ Sep 4, 2020 at 19:03
  • 1
    $\begingroup$ On the contrary, you are still fitting an intercept -- but you just aren't estimating it! $\endgroup$
    – whuber
    Sep 4, 2020 at 19:09
  • 2
    $\begingroup$ @student010101 Sure you are. You will just be estimating it as zero. Whether you are fitting an intercept or not is simply a matter of whether you exclude it or include it in the design matrix. $\endgroup$ Sep 4, 2020 at 19:11
  • 3
    $\begingroup$ @student010101 are you sure you get the same coefficient for X2 ? $\endgroup$ Sep 4, 2020 at 19:17
  • 2
    $\begingroup$ Could you ask a seperate question about that ? I've tried to answer the question as posed, but comments aren't for extended discussion. It's a good question and worthy of a question in it's own right. Also, you will get better answers that way (provided it's not a duplicate) $\endgroup$ Sep 4, 2020 at 19:43
4
$\begingroup$

For what you said to be true all the time, your predictors should not only be orthogonal to each other, but they must also be orthogonal to the vector of 1s.

What this means is you need to look at the centered version of each predictor (when you center a vector, you are orthogonalizing it with respect to the vector of 1s), and assess orthogonality (in this case, you're assessing uncorrelatedness since uncorrelated means the same thing as orthogonality for centered vectors/predictors).

Note also that $\langle X_1, X_2 \rangle = 0$ does NOT necessarily mean $\langle X_1 - \bar{X_1}, X_2 - \bar{X_2} \rangle = 0$, as you can observe in your case. If they are, then you can individually regress $y$ on $1, X_1 - \bar{X}_1, X_2 - \bar{X}_2$ INDIVIDUALLY and obtain $\hat{\beta}_0, \hat{\beta}_1, \hat{\beta}_2$, which will be equivalent to the coefficients determined from regressing on all 3 variables simultaneously. In addition the $\hat{\beta}_1, \hat{\beta}_2$ will be equivalent to the coefficients determined from multiple regression of $y$ on $1, X_1, X_2$ (uncentered versions), but $\hat{\beta}_0$ will be different.

This is easy to see when you examine $$ \hat{\beta} = (X^TX)^{-1}X^TY $$

If $X_1 \perp X_2 \perp 1$, then the columns of $X$ are orthogonal, and hence $X^TX$ is a diagonal matrix. The inverse of a diagonal matrix is diagonal. So each coefficient will have no influence on another.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.