Suppose there is a regression with three variables $X_1, X_2, X_3$. Say $X_1$ and $X_2$ have near-multicollinearity, but $X_3$ is (nearly) orthogonal to both.
Will $\beta_3$ experience the same problems the other two coefficients?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
No, $\beta_3$ is not affected by the collinearity of $X_1$ and $X_2$. In fact, $X_3$ doesn't even have to be nearly orthogonal, as long as it is not collinear with $X_1$ and $X_2$.
A figure might help:
First recall that notions like "collinearity" and "orthogonality" refer to the column vectors $\mathbf x_1, \mathbf x_2, \mathbf x_3$ of the design matrix $D = (\mathbf x_1, \mathbf x_2, \mathbf x_3)$. Now, let's call the response variable $Y$ and consider the situation where we have only two measurements. This is only so that it better corresponds to the figure, the arguments hold for an arbitrary number $n$ of measurements; see also the ending remark of this post.
In the figure, note that $\mathbf x_1$ and $\mathbf x_2$ are collinear, while $\mathbf x_3$ is neither collinear with $\mathbf x_1$ and $\mathbf x_2$ nor orthogonal to them. To approximate $\mathbf y$ with $\mathbf x_1, \mathbf x_2$, and $\mathbf x_3$, we have to create $\mathbf z = \beta_1\mathbf x_1 + \beta_2 \mathbf x_2$ with a linear combination of $\mathbf x_1$ and $\mathbf x_2$, and this $\mathbf z$ has to be such that, when adding an appropriate multiple of $\mathbf x_3$, we will reach $\mathbf y$: $\beta \mathbf x_3 = \mathbf y - \mathbf z$. This condition makes $\mathbf z$ unique.
Now, while there are, because of colinearity, infinitely many possibilities for $\beta_1$ and $\beta_2$ to create $\mathbf z$ from $\mathbf x_1$ and $\mathbf x_2$, there is only one multiple of $\mathbf x_3$ that, when added to $\mathbf z$, gives $\mathbf y$. Thus, $\beta_3$ is not affected by the collinearity.
If we have more than two measurements, it is probably the case that $\mathbf y$ will not be an element of the two-dimensional subplane $\mathcal P$ spanned by $\mathbf x_1, \mathbf x_2$, and $\mathbf x_3$. In this case, the above figure depicts this subplane $\mathcal P$ and the $\mathbf y$ in the figure will be the projection of the proper $\mathbf y$ to $\mathcal P$.
