I'm not sure what this is called but I remember seeing a colleague of mine doing a multivariate regression much like

$$Y \sim X_1 + X_2 + X_3$$

and then he said he would "orthogonalize" so he regressed each of the IVs against the other ones, and took the residuals..as in:

$$Z_1 = residuals(X_1 \sim X_2 + X_3)$$ $$Z_2 = residuals(X_2 \sim X_1 + X_3)$$ $$Z_3 = residuals(X_3 \sim X_1 + X_2)$$

after which he redid the regression:

$$Y \sim Z_1 + Z_2 + Z_3$$

I have never seen this before, is there a reference or name under which I can search read more about this and why/when would one want to do something like this?


I think you misremember the end of the process. In R, it would go like this:

# generating random x1 x2 x3 in (0,1) (10 values each)
> x1 <- runif(10)
> x2 <- runif(10)
> x3 <- runif(10)

# generating y
> y <- x1 + 2*x2 + 3*x3 + rnorm(10)

# classical regression
> lm(y ~ x1 + x2 + x3)

lm(formula = y ~ x1 + x2 + x3)

(Intercept)           x1           x2           x3  
 0.2270       2.0088       0.2746       3.1529  

# "orthogonalized" regression
> lm(x1 ~ x2 + x3)$residuals -> z1
> lm(x2 ~ x1 + x3)$residuals -> z2
> lm(x3 ~ x1 + x2)$residuals -> z3

> lm(y ~ z1) 

lm(formula = y ~ z1)

(Intercept)           z1  
      3.056        2.009  

> lm(y ~ z2)

lm(formula = y ~ z2)

(Intercept)           z2  
     3.0560       0.2746  

> lm(y ~ z3)

lm(formula = y ~ z3)

(Intercept)           z3  
      3.056        3.153  

See? You get the same estimates $\hat \beta_i$ for $i = 1,2,3$. Note that the intercepts are differents; the residual $z_i$ are centered so the intercept of eg the regression y ~ z1 is just the mean of $y$ (and similarly for $z_2$, $z_3$). Once you get the $\hat \beta_i$ it is not difficult to find the intercept of the classical regression.

Mathematical explications will be find in page 54-55 of last edition of The elements of statiscal learning — much clearer and accurate that anything I could write (available on line).

  • 1
    $\begingroup$ A pedantic comment: you meant the coefficient estimates, and you meant those of the non-constant explanatory variables. The intercept estimate is different between the full regression and the orthogonalized regression, but it is probably of little importance. $\endgroup$ – StasK Mar 2 '12 at 21:07
  • $\begingroup$ Thanks Elvis, very interesting. We do end up getting the same coefficient estimates, however, so I wonder why people would orthogonalize a design matrix? To what avail other than perhaps computational? $\endgroup$ – Palace Chan Mar 5 '12 at 15:57
  • $\begingroup$ Please note that $z_1$ is orthogonal to $x_2$ and $x_3$, but not to $z_2$ and $z_3$... I don’t know if there is any interest to this procedure, besides the obvious pedagogical interest: it allows to understand the behavior of regression when covariables aren’t orthogonal... One might suspect some advantage in term of numeric stability, for example. May be you should start a new question on this matter. $\endgroup$ – Elvis Mar 5 '12 at 21:17

This is the Frisch Waugh Lovell theorem in action


Ruud's An Introduction to Classical Econometric Theory rides that FWL pony about as far as possible. It's a really interesting geometric take on regression.

  • $\begingroup$ Can you give us a short description of what he does with it? $\endgroup$ – Elvis Mar 5 '12 at 21:18
  • $\begingroup$ Mathematical projection is the primary theoretical principle for the book. I've seen the the geometry of OLS before in other sources, but he also uses it to introduce partitioned regression, conditional expectations, generalized least squares, instrumental variables, and relative efficiency of estimators. $\endgroup$ – Dimitriy V. Masterov Mar 6 '12 at 14:54
  • $\begingroup$ This might be a nice example to see if you might like the approach in the book: elsa.berkeley.edu/GMTheorem/index.html. $\endgroup$ – Dimitriy V. Masterov Mar 7 '12 at 0:06
  • $\begingroup$ @Dimitriy, the links are dead, could you update them or suggest another intro to geometry + regression ? thanks $\endgroup$ – denis May 25 '16 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.