Orthogonalized regression reference?

Question

I'm not sure what this is called but I remember seeing a colleague of mine doing a multivariate regression much like

$$Y \sim X_1 + X_2 + X_3$$

and then he said he would "orthogonalize" so he regressed each of the IVs against the other ones, and took the residuals..as in:

$$Z_1 = residuals(X_1 \sim X_2 + X_3)$$ $$Z_2 = residuals(X_2 \sim X_1 + X_3)$$ $$Z_3 = residuals(X_3 \sim X_1 + X_2)$$

after which he redid the regression:

$$Y \sim Z_1 + Z_2 + Z_3$$

I have never seen this before, is there a reference or name under which I can search read more about this and why/when would one want to do something like this?

Answer 1

I think you misremember the end of the process. In R, it would go like this:

# generating random x1 x2 x3 in (0,1) (10 values each)
> x1 <- runif(10)
> x2 <- runif(10)
> x3 <- runif(10)

# generating y
> y <- x1 + 2*x2 + 3*x3 + rnorm(10)

# classical regression
> lm(y ~ x1 + x2 + x3)

Call:
lm(formula = y ~ x1 + x2 + x3)

Coefficients:
(Intercept)           x1           x2           x3  
 0.2270       2.0088       0.2746       3.1529  


# "orthogonalized" regression
> lm(x1 ~ x2 + x3)$residuals -> z1
> lm(x2 ~ x1 + x3)$residuals -> z2
> lm(x3 ~ x1 + x2)$residuals -> z3

> lm(y ~ z1) 

Call:
lm(formula = y ~ z1)

Coefficients:
(Intercept)           z1  
      3.056        2.009  

> lm(y ~ z2)

Call:
lm(formula = y ~ z2)

Coefficients:
(Intercept)           z2  
     3.0560       0.2746  

> lm(y ~ z3)

Call:
lm(formula = y ~ z3)

Coefficients:
(Intercept)           z3  
      3.056        3.153  

See? You get the same estimates $\hat \beta_i$ for $i = 1,2,3$. Note that the intercepts are differents; the residual $z_i$ are centered so the intercept of eg the regression y ~ z1 is just the mean of $y$ (and similarly for $z_2$, $z_3$). Once you get the $\hat \beta_i$ it is not difficult to find the intercept of the classical regression.

Mathematical explications will be find in page 54-55 of last edition of The elements of statiscal learning — much clearer and accurate that anything I could write (available on line).

Answer 2

This is the Frisch Waugh Lovell theorem in action

Answer 3

Ruud's econometrics text book rides that FWL pony about as far as possible: http://elsa.berkeley.edu/~ruud/cet/index.html. It's a really interesting geometric take on regression.