Multicollinearity in simple linear regression

Question

If there's perfect or near multicollinearity problem in a simple linear regression $y_i = a + b x_i + u_i$, what characteristics does $x_i$ have?

I think if there's perfect multicollinearity, it means that $x_i$ is simply a constant.

But if there's near multicollinearity, what happens to $x_i$?

Answer 1

$\newcommand{\e}{\varepsilon}$$\newcommand{\one}{\mathbf 1}$$\newcommand{\x}{\mathbf x}$If we're doing simple linear regression we can view this as $y = X\beta + \e$ with $$ X = \left[\begin{array}{cc}1 & x_1 \\ \vdots & \vdots \\ 1 & x_n\end{array}\right] = [\one \; \x] \in \mathbb R^{n\times p}. $$ This already makes it clear that perfect multicollinearity happens when $\x \in \text{span}(\one)$, i.e. $\x$ is constant, but we can say more than this.

To get $\hat\beta$ we need $$ X^TX = \left[\begin{array}{cc}n & n\bar x \\ n\bar x & \x^T\x\end{array}\right] $$ which we need to be invertible, which happens if and only if $\det X^TX \neq 0$. We have $$ \det X^TX = n\x^T\x - n^2\bar x^2 = n(\x^T\x - n\bar x^2) $$ and it's well-known that $$ \x^T\x - n\bar x^2 = \sum_{i=1}^n (x_i - \bar x)^2 $$ so $$ \det X^TX > 0\iff\sum_{i=1}^n (x_i - \bar x)^2 > 0. $$ If $\x \in \text{span}(\one)$ then $\det X^TX = 0$, and this also shows that the smaller the sample variance of $\x$ is, the more ill-conditioned this regression is.

Furthermore, under the assumption that $\text E(\e) = \mathbf0 $ and $\text E(\e\e^T) = \sigma^2 I$ we know $$ \text{Var}(\hat\beta) = \sigma^2 (X^TX)^{-1} $$ and $X^TX$ is 2x2 so we can work out that $$ (X^TX)^{-1} = \frac{1}{\det X^TX} \left[\begin{array}{cc}\x^T\x & -n\bar x \\ -n\bar x & n\end{array}\right] $$ which shows that the standard errors will also increase as $\x$ gets closer and closer to being in $\text{span}(\one)$, as measured by the sample variance of $\x$. This is just a consequence of the usual result that multicollinearity inflates standard errors but it's nice that we can so explicitly see how it does it here.