Orthogonalized regression reference?

Question

I'm not sure what this is called but I remember seeing a colleague of mine doing a multivariate regression much like

$$Y \sim X_1 + X_2 + X_3$$

and then he said he would "orthogonalize" so he regressed each of the IVs against the other ones, and took the residuals..as in:

$$Z_1 = \text{residuals}(X_1 \sim X_2 + X_3)$$ $$Z_2 = \text{residuals}(X_2 \sim X_1 + X_3)$$ $$Z_3 = \text{residuals}(X_3 \sim X_1 + X_2)$$

after which he redid the regression:

$$Y \sim Z_1 + Z_2 + Z_3$$

I have never seen this before, is there a reference or name under which I can search read more about this and why/when would one want to do something like this?

Answer 1

I think you misremember the end of the process. In R, it would go like this:

# generating random x1 x2 x3 in (0,1) (10 values each)
> x1 <- runif(10)
> x2 <- runif(10)
> x3 <- runif(10)

# generating y
> y <- x1 + 2*x2 + 3*x3 + rnorm(10)

# classical regression
> lm(y ~ x1 + x2 + x3)

Call:
lm(formula = y ~ x1 + x2 + x3)

Coefficients:
(Intercept)           x1           x2           x3  
 0.2270       2.0088       0.2746       3.1529  


# "orthogonalized" regression
> lm(x1 ~ x2 + x3)$residuals -> z1
> lm(x2 ~ x1 + x3)$residuals -> z2
> lm(x3 ~ x1 + x2)$residuals -> z3

> lm(y ~ z1) 

Call:
lm(formula = y ~ z1)

Coefficients:
(Intercept)           z1  
      3.056        2.009  

> lm(y ~ z2)

Call:
lm(formula = y ~ z2)

Coefficients:
(Intercept)           z2  
     3.0560       0.2746  

> lm(y ~ z3)

Call:
lm(formula = y ~ z3)

Coefficients:
(Intercept)           z3  
      3.056        3.153  

See? You get the same estimates $\hat \beta_i$ for $i = 1,2,3$. Note that the intercepts are differents; the residual $z_i$ are centered so the intercept of eg the regression y ~ z1 is just the mean of $y$ (and similarly for $z_2$, $z_3$). Once you get the $\hat \beta_i$ it is not difficult to find the intercept of the classical regression.

Mathematical explications will be find in page 54-55 of last edition of The elements of statiscal learning — much clearer and accurate that anything I could write (available on line).

Answer 2

This is the Frisch Waugh Lovell theorem in action

Answer 3

Ruud's An Introduction to Classical Econometric Theory rides that FWL pony about as far as possible. It's a really interesting geometric take on regression.

Answer 4

Model can be reparametrized in such a way that two new likelihood equations emerge, each with just one unknown parameter. This will facilitate solving the likelihood equations and also help the general interpretation and use of regression models. (7.2.2 in [hendry2007econometric])

Suppose you want to reparametrize the following model: (note that $X_{3}$ can be any transformation of some previous regressor)

$$ Y \sim X_{1} + X_{2} + X_{3} $$

$X_{1}$, $X_{2}$ and $X_{3}$ can be orthogonalized at the same time. In the book, the operation is based on a constant vector.

$$ \begin{aligned} Z_{1} &= \mathrm{residuals}\left(X_{1} \sim 1 \right) \\ Z_{2} &= \mathrm{residuals}\left(X_{2} \sim 1 + X_{1} \right) \\ Z_{3} &= \mathrm{residuals}\left(X_{3} \sim 1 + X_{1} + X_{2}\right) \end{aligned} $$

Following the example by @Elvis:

library(magrittr)

## generating random x1 x2 x3 in (0,1) (10 values each)
x1 <- runif(10)
x2 <- runif(10)
x3 <- runif(10)

## generating y
y <- x1 + 2 * x2 + 3 * x3 + rnorm(10)

## classical regression
lm(y ~ x1 + x2 + x3) %>% summary()

## orthogonalize regressors on a unit vector
lm(x1 ~ 1)$residuals -> z1
lm(x2 ~ 1 + x1)$residuals -> z2
lm(x3 ~ 1 + x1 + x2)$residuals -> z3

lm(y ~ z1 + z2 + z3) %>% summary()

You will have:

Call:
lm(formula = y ~ x1 + x2 + x3)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -2.1528     0.7973  -2.700  0.03558 *  
x1            2.1005     0.9730   2.159  0.07421 .  
x2            0.7895     0.9364   0.843  0.43149    
x3            6.8008     1.0055   6.764  0.00051 ***

Residual standard error: 0.7628 on 6 degrees of freedom
Multiple R-squared:  0.9293,    Adjusted R-squared:  0.8939 
F-statistic: 26.27 on 3 and 6 DF,  p-value: 0.0007538

Call:
lm(formula = y ~ z1 + z2 + z3)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  3.18106    0.24121  13.188 1.17e-05 ***
z1          -0.05549    0.72386  -0.077  0.94139    
z2           4.41463    0.76784   5.749  0.00121 ** 
z3           6.80079    1.00551   6.764  0.00051 ***

Residual standard error: 0.7628 on 6 degrees of freedom
Multiple R-squared:  0.9293,    Adjusted R-squared:  0.8939 
F-statistic: 26.27 on 3 and 6 DF,  p-value: 0.0007538

So the intercept in the second model can be interpreted as the expected value for an individual with average values of x1, x2 and x3, and its standard error is reduced by 78.21%. Most of the time, you are very interested in this value.

Also, maximum likelihood estimators become much easier to handle. (5.2.3 in [hendry2007econometric])

Referece

• hendry2007econometric Hendry, D. F., & Nielsen, B. (2007). Econometric modeling: a likelihood approach. Princeton University Press.