A regression in which all the IVs are replaced by their residuals after regressing them with the rest of the IVs. I have seen (for example in Elements of Statistical Learning) how doing this in a multivariate regression yields the multivariate coefficients from univariate regressions. But other than this, why would one benefit from using orthogonalized IVs rather than the regular ones? What are some reasons to be doing this in a multivariate regression?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
|
The key insight to understand this is that correlation is somehow equivalent to predictability with linear functions. Imagine you have two (mean zero) random variables $X$ and $Y$ and you try to predict $Y$ from $X$. The optimal linear predictor (in a least squares sense) is $$y = \frac{\sigma_{yx}}{\sigma_x^2}x = \frac{\rho_{yx} \sigma_y}{\sigma_x}x,$$ which basically means that you divide $x$ by the variance of $X$ multiply it with the correlation coefficient and rescale it with the variance of $Y$. If the correlation is $\rho_{yx}=0$, then there's no meaningful prediction of $Y$ from $X$ (except $Y$ is zero all the time). On the other hand, if you have to uncorrelated (mean zero) random variables $X$ and $Y$ and you project them on two linearly dependent vectors $\mathbf w$ and $\mathbf v$, then the results will be correlated: Assume that $\|\mathbf v\|=\|\mathbf w\|=1$, $W=(\mathbf v,\mathbf w)^\top$, $\mathbf z = (x,y)^\top$, and $\tilde{\mathbf z} = W \mathbf z$, then $$\Sigma_{\tilde z\tilde z} = E\left[\tilde{\mathbf z} \tilde{\mathbf z}^\top \right] = WE\left[\mathbf z \mathbf z^\top \right]W^\top = WDW^\top \not= I$$ where $D$ is a diagonal matrix (the covariance matrix of $X$ and $Y$). However, you can also see that $\tilde{Z_1}$ and $\tilde{Z_2}$ would still have been uncorrelated if $\mathbf v$ and $\mathbf w$ were orthogonal and $D=I$. If you have several IVs and want to do linear regression, you want your IVs be uncorrelated because then you can use them one at a time. If they are uncorrelated, then you cannot gain information about one IV from another through a linear function, so-in some sense-each of the uncorrelated IVs gives you a fresh look on the dependent variable. Since linear dependence means correlation, you use orthogonalized IVs in regression. In general, however, orthogonalization is not enough to get decorrelation. For that, you must use the principal components (from PCA) which are an orthogonal system. If you additionally equalize the variances in the PCA system (which is called whitening then), you can rotate the IVs again, since they will stay uncorrelated then (but only after whitening and only for orthogonal transforms). In summary, decorrelation means removing linear predictability between the IVs. |
|||||
|
