5
$\begingroup$

I'm not sure what this is called but I remember seeing a colleague of mine doing a multivariate regression much like

$$Y \sim X_1 + X_2 + X_3$$

and then he said he would "orthogonalize" so he regressed each of the IVs against the other ones, and took the residuals..as in:

$$Z_1 = \text{residuals}(X_1 \sim X_2 + X_3)$$ $$Z_2 = \text{residuals}(X_2 \sim X_1 + X_3)$$ $$Z_3 = \text{residuals}(X_3 \sim X_1 + X_2)$$

after which he redid the regression:

$$Y \sim Z_1 + Z_2 + Z_3$$

I have never seen this before, is there a reference or name under which I can search read more about this and why/when would one want to do something like this?

$\endgroup$

4 Answers 4

8
$\begingroup$

I think you misremember the end of the process. In R, it would go like this:

# generating random x1 x2 x3 in (0,1) (10 values each)
> x1 <- runif(10)
> x2 <- runif(10)
> x3 <- runif(10)

# generating y
> y <- x1 + 2*x2 + 3*x3 + rnorm(10)

# classical regression
> lm(y ~ x1 + x2 + x3)

Call:
lm(formula = y ~ x1 + x2 + x3)

Coefficients:
(Intercept)           x1           x2           x3  
 0.2270       2.0088       0.2746       3.1529  


# "orthogonalized" regression
> lm(x1 ~ x2 + x3)$residuals -> z1
> lm(x2 ~ x1 + x3)$residuals -> z2
> lm(x3 ~ x1 + x2)$residuals -> z3

> lm(y ~ z1) 

Call:
lm(formula = y ~ z1)

Coefficients:
(Intercept)           z1  
      3.056        2.009  

> lm(y ~ z2)

Call:
lm(formula = y ~ z2)

Coefficients:
(Intercept)           z2  
     3.0560       0.2746  

> lm(y ~ z3)

Call:
lm(formula = y ~ z3)

Coefficients:
(Intercept)           z3  
      3.056        3.153  

See? You get the same estimates $\hat \beta_i$ for $i = 1,2,3$. Note that the intercepts are differents; the residual $z_i$ are centered so the intercept of eg the regression y ~ z1 is just the mean of $y$ (and similarly for $z_2$, $z_3$). Once you get the $\hat \beta_i$ it is not difficult to find the intercept of the classical regression.

Mathematical explications will be find in page 54-55 of last edition of The elements of statiscal learning — much clearer and accurate that anything I could write (available on line).

$\endgroup$
3
  • 1
    $\begingroup$ A pedantic comment: you meant the coefficient estimates, and you meant those of the non-constant explanatory variables. The intercept estimate is different between the full regression and the orthogonalized regression, but it is probably of little importance. $\endgroup$
    – StasK
    Mar 2, 2012 at 21:07
  • $\begingroup$ Thanks Elvis, very interesting. We do end up getting the same coefficient estimates, however, so I wonder why people would orthogonalize a design matrix? To what avail other than perhaps computational? $\endgroup$ Mar 5, 2012 at 15:57
  • $\begingroup$ Please note that $z_1$ is orthogonal to $x_2$ and $x_3$, but not to $z_2$ and $z_3$... I don’t know if there is any interest to this procedure, besides the obvious pedagogical interest: it allows to understand the behavior of regression when covariables aren’t orthogonal... One might suspect some advantage in term of numeric stability, for example. May be you should start a new question on this matter. $\endgroup$
    – Elvis
    Mar 5, 2012 at 21:17
6
$\begingroup$

This is the Frisch Waugh Lovell theorem in action

$\endgroup$
1
$\begingroup$

Ruud's An Introduction to Classical Econometric Theory rides that FWL pony about as far as possible. It's a really interesting geometric take on regression.

$\endgroup$
4
  • $\begingroup$ Can you give us a short description of what he does with it? $\endgroup$
    – Elvis
    Mar 5, 2012 at 21:18
  • $\begingroup$ Mathematical projection is the primary theoretical principle for the book. I've seen the the geometry of OLS before in other sources, but he also uses it to introduce partitioned regression, conditional expectations, generalized least squares, instrumental variables, and relative efficiency of estimators. $\endgroup$
    – dimitriy
    Mar 6, 2012 at 14:54
  • $\begingroup$ This might be a nice example to see if you might like the approach in the book: elsa.berkeley.edu/GMTheorem/index.html. $\endgroup$
    – dimitriy
    Mar 7, 2012 at 0:06
  • $\begingroup$ @Dimitriy, the links are dead, could you update them or suggest another intro to geometry + regression ? thanks $\endgroup$
    – denis
    May 25, 2016 at 16:37
0
$\begingroup$

Model can be reparametrized in such a way that two new likelihood equations emerge, each with just one unknown parameter. This will facilitate solving the likelihood equations and also help the general interpretation and use of regression models. (7.2.2 in [hendry2007econometric])

Suppose you want to reparametrize the following model: (note that $X_{3}$ can be any transformation of some previous regressor)

$$ Y \sim X_{1} + X_{2} + X_{3} $$

$X_{1}$, $X_{2}$ and $X_{3}$ can be orthogonalized at the same time. In the book, the operation is based on a constant vector.

$$ \begin{aligned} Z_{1} &= \mathrm{residuals}\left(X_{1} \sim 1 \right) \\ Z_{2} &= \mathrm{residuals}\left(X_{2} \sim 1 + X_{1} \right) \\ Z_{3} &= \mathrm{residuals}\left(X_{3} \sim 1 + X_{1} + X_{2}\right) \end{aligned} $$

Following the example by @Elvis:

library(magrittr)

## generating random x1 x2 x3 in (0,1) (10 values each)
x1 <- runif(10)
x2 <- runif(10)
x3 <- runif(10)

## generating y
y <- x1 + 2 * x2 + 3 * x3 + rnorm(10)

## classical regression
lm(y ~ x1 + x2 + x3) %>% summary()

## orthogonalize regressors on a unit vector
lm(x1 ~ 1)$residuals -> z1
lm(x2 ~ 1 + x1)$residuals -> z2
lm(x3 ~ 1 + x1 + x2)$residuals -> z3

lm(y ~ z1 + z2 + z3) %>% summary()

You will have:

Call:
lm(formula = y ~ x1 + x2 + x3)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -2.1528     0.7973  -2.700  0.03558 *  
x1            2.1005     0.9730   2.159  0.07421 .  
x2            0.7895     0.9364   0.843  0.43149    
x3            6.8008     1.0055   6.764  0.00051 ***

Residual standard error: 0.7628 on 6 degrees of freedom
Multiple R-squared:  0.9293,    Adjusted R-squared:  0.8939 
F-statistic: 26.27 on 3 and 6 DF,  p-value: 0.0007538

Call:
lm(formula = y ~ z1 + z2 + z3)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  3.18106    0.24121  13.188 1.17e-05 ***
z1          -0.05549    0.72386  -0.077  0.94139    
z2           4.41463    0.76784   5.749  0.00121 ** 
z3           6.80079    1.00551   6.764  0.00051 ***

Residual standard error: 0.7628 on 6 degrees of freedom
Multiple R-squared:  0.9293,    Adjusted R-squared:  0.8939 
F-statistic: 26.27 on 3 and 6 DF,  p-value: 0.0007538

So the intercept in the second model can be interpreted as the expected value for an individual with average values of x1, x2 and x3, and its standard error is reduced by 78.21%. Most of the time, you are very interested in this value.

Also, maximum likelihood estimators become much easier to handle. (5.2.3 in [hendry2007econometric])

Referece

  • hendry2007econometric Hendry, D. F., & Nielsen, B. (2007). Econometric modeling: a likelihood approach. Princeton University Press.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.