# Why would one want to carry out an orthogonalized (gram schmit) regression?

A regression in which all the IVs are replaced by their residuals after regressing them with the rest of the IVs. I have seen (for example in Elements of Statistical Learning) how doing this in a multivariate regression yields the multivariate coefficients from univariate regressions. But other than this, why would one benefit from using orthogonalized IVs rather than the regular ones? What are some reasons to be doing this in a multivariate regression?

-

The key insight to understand this is that correlation is somehow equivalent to predictability with linear functions.

Imagine you have two (mean zero) random variables $X$ and $Y$ and you try to predict $Y$ from $X$. The optimal linear predictor (in a least squares sense) is $$y = \frac{\sigma_{yx}}{\sigma_x^2}x = \frac{\rho_{yx} \sigma_y}{\sigma_x}x,$$ which basically means that you divide $x$ by the variance of $X$ multiply it with the correlation coefficient and rescale it with the variance of $Y$. If the correlation is $\rho_{yx}=0$, then there's no meaningful prediction of $Y$ from $X$ (except $Y$ is zero all the time).

On the other hand, if you have to uncorrelated (mean zero) random variables $X$ and $Y$ and you project them on two linearly dependent vectors $\mathbf w$ and $\mathbf v$, then the results will be correlated: Assume that $\|\mathbf v\|=\|\mathbf w\|=1$, $W=(\mathbf v,\mathbf w)^\top$, $\mathbf z = (x,y)^\top$, and $\tilde{\mathbf z} = W \mathbf z$, then $$\Sigma_{\tilde z\tilde z} = E\left[\tilde{\mathbf z} \tilde{\mathbf z}^\top \right] = WE\left[\mathbf z \mathbf z^\top \right]W^\top = WDW^\top \not= I$$ where $D$ is a diagonal matrix (the covariance matrix of $X$ and $Y$).

However, you can also see that $\tilde{Z_1}$ and $\tilde{Z_2}$ would still have been uncorrelated if $\mathbf v$ and $\mathbf w$ were orthogonal and $D=I$.

If you have several IVs and want to do linear regression, you want your IVs be uncorrelated because then you can use them one at a time. If they are uncorrelated, then you cannot gain information about one IV from another through a linear function, so-in some sense-each of the uncorrelated IVs gives you a fresh look on the dependent variable. Since linear dependence means correlation, you use orthogonalized IVs in regression. In general, however, orthogonalization is not enough to get decorrelation. For that, you must use the principal components (from PCA) which are an orthogonal system. If you additionally equalize the variances in the PCA system (which is called whitening then), you can rotate the IVs again, since they will stay uncorrelated then (but only after whitening and only for orthogonal transforms).

In summary, decorrelation means removing linear predictability between the IVs.

-
It makes sense to me that you can use them one at a time if the IVs are orthogonal. But, while their subspace remains the same, the basis changed so how do you interpret the new coefficients of an orthogonalized family of IVs? –  Palace Chan Mar 9 '12 at 23:06
As a linear combination of the old one. For example, if you run PCA on color pixels (RGB) of natural images, one component that you get is usually aroun $3^{-1/2} (1,1,1)$. When you think about it, this new components represents the dark-light feature, because it does not favor any color and the brighter the pixel is the larger the projection on the component will be. –  fabee Mar 10 '12 at 9:00