2
$\begingroup$

Suppose there is a regression with three variables $X_1, X_2, X_3$. Say $X_1$ and $X_2$ have near-multicollinearity, but $X_3$ is (nearly) orthogonal to both.

Will $\beta_3$ experience the same problems the other two coefficients?

$\endgroup$

1 Answer 1

3
$\begingroup$

No, $\beta_3$ is not affected by the collinearity of $X_1$ and $X_2$. In fact, $X_3$ doesn't even have to be nearly orthogonal, as long as it is not collinear with $X_1$ and $X_2$.

A figure might help:
First recall that notions like "collinearity" and "orthogonality" refer to the column vectors $\mathbf x_1, \mathbf x_2, \mathbf x_3$ of the design matrix $D = (\mathbf x_1, \mathbf x_2, \mathbf x_3)$. Now, let's call the response variable $Y$ and consider the situation where we have only two measurements. This is only so that it better corresponds to the figure, the arguments hold for an arbitrary number $n$ of measurements; see also the ending remark of this post.

enter image description here

In the figure, note that $\mathbf x_1$ and $\mathbf x_2$ are collinear, while $\mathbf x_3$ is neither collinear with $\mathbf x_1$ and $\mathbf x_2$ nor orthogonal to them. To approximate $\mathbf y$ with $\mathbf x_1, \mathbf x_2$, and $\mathbf x_3$, we have to create $\mathbf z = \beta_1\mathbf x_1 + \beta_2 \mathbf x_2$ with a linear combination of $\mathbf x_1$ and $\mathbf x_2$, and this $\mathbf z$ has to be such that, when adding an appropriate multiple of $\mathbf x_3$, we will reach $\mathbf y$: $\beta \mathbf x_3 = \mathbf y - \mathbf z$. This condition makes $\mathbf z$ unique.

Now, while there are, because of colinearity, infinitely many possibilities for $\beta_1$ and $\beta_2$ to create $\mathbf z$ from $\mathbf x_1$ and $\mathbf x_2$, there is only one multiple of $\mathbf x_3$ that, when added to $\mathbf z$, gives $\mathbf y$. Thus, $\beta_3$ is not affected by the collinearity.

If we have more than two measurements, it is probably the case that $\mathbf y$ will not be an element of the two-dimensional subplane $\mathcal P$ spanned by $\mathbf x_1, \mathbf x_2$, and $\mathbf x_3$. In this case, the above figure depicts this subplane $\mathcal P$ and the $\mathbf y$ in the figure will be the projection of the proper $\mathbf y$ to $\mathcal P$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.