# Orthogonalized regression reference?

I'm not sure what this is called but I remember seeing a colleague of mine doing a multivariate regression much like

$$Y \sim X_1 + X_2 + X_3$$

and then he said he would "orthogonalize" so he regressed each of the IVs against the other ones, and took the residuals..as in:

$$Z_1 = residuals(X_1 \sim X_2 + X_3)$$ $$Z_2 = residuals(X_2 \sim X_1 + X_3)$$ $$Z_3 = residuals(X_3 \sim X_1 + X_2)$$

after which he redid the regression:

$$Y \sim Z_1 + Z_2 + Z_3$$

I have never seen this before, is there a reference or name under which I can search read more about this and why/when would one want to do something like this?

-

I think you misremember the end of the process. In R, it would go like this:

# generating random x1 x2 x3 in (0,1) (10 values each)
> x1 <- runif(10)
> x2 <- runif(10)
> x3 <- runif(10)

# generating y
> y <- x1 + 2*x2 + 3*x3 + rnorm(10)

# classical regression
> lm(y ~ x1 + x2 + x3)

Call:
lm(formula = y ~ x1 + x2 + x3)

Coefficients:
(Intercept)           x1           x2           x3
0.2270       2.0088       0.2746       3.1529

# "orthogonalized" regression
> lm(x1 ~ x2 + x3)$residuals -> z1 > lm(x2 ~ x1 + x3)$residuals -> z2
> lm(x3 ~ x1 + x2)$residuals -> z3 > lm(y ~ z1) Call: lm(formula = y ~ z1) Coefficients: (Intercept) z1 3.056 2.009 > lm(y ~ z2) Call: lm(formula = y ~ z2) Coefficients: (Intercept) z2 3.0560 0.2746 > lm(y ~ z3) Call: lm(formula = y ~ z3) Coefficients: (Intercept) z3 3.056 3.153 See? You get the same estimates$\hat \beta_i$for$i = 1,2,3$. Note that the intercepts are differents; the residual$z_i$are centered so the intercept of eg the regression y ~ z1 is just the mean of$y$(and similarly for$z_2$,$z_3$). Once you get the$\hat \beta_i$it is not difficult to find the intercept of the classical regression. Mathematical explications will be find in page 54-55 of last edition of The elements of statiscal learning — much clearer and accurate that anything I could write (available on line). - A pedantic comment: you meant the coefficient estimates, and you meant those of the non-constant explanatory variables. The intercept estimate is different between the full regression and the orthogonalized regression, but it is probably of little importance. – StasK Mar 2 '12 at 21:07 Corrected. Thanks. – Elvis Mar 2 '12 at 21:30 Thanks Elvis, very interesting. We do end up getting the same coefficient estimates, however, so I wonder why people would orthogonalize a design matrix? To what avail other than perhaps computational? – Palace Chan Mar 5 '12 at 15:57 Please note that$z_1$is orthogonal to$x_2$and$x_3$, but not to$z_2$and$z_3\$... I don’t know if there is any interest to this procedure, besides the obvious pedagogical interest: it allows to understand the behavior of regression when covariables aren’t orthogonal... One might suspect some advantage in term of numeric stability, for example. May be you should start a new question on this matter. –  Elvis Mar 5 '12 at 21:17

This is the Frisch Waugh Lovell theorem in action

-