If there's perfect or near multicollinearity problem in a simple linear regression $y_i = a + b x_i + u_i$, what characteristics does $x_i$ have?

I think if there's perfect multicollinearity, it means that $x_i$ is simply a constant.

But if there's near multicollinearity, what happens to $x_i$?

  • 1
    $\begingroup$ The one of the assumption of the bivariate regression is that there should be some variability in X (ie non constant). If X is constant then you can not identify your parameters ie a and b. $\endgroup$
    – Neeraj
    Jun 11, 2019 at 6:47
  • 2
    $\begingroup$ What does multicolinearity mean in the case of simple regression? $\endgroup$
    – David
    Jun 11, 2019 at 7:29

1 Answer 1


$\newcommand{\e}{\varepsilon}$$\newcommand{\one}{\mathbf 1}$$\newcommand{\x}{\mathbf x}$If we're doing simple linear regression we can view this as $y = X\beta + \e$ with $$ X = \left[\begin{array}{cc}1 & x_1 \\ \vdots & \vdots \\ 1 & x_n\end{array}\right] = [\one \; \x] \in \mathbb R^{n\times p}. $$ This already makes it clear that perfect multicollinearity happens when $\x \in \text{span}(\one)$, i.e. $\x$ is constant, but we can say more than this.

To get $\hat\beta$ we need $$ X^TX = \left[\begin{array}{cc}n & n\bar x \\ n\bar x & \x^T\x\end{array}\right] $$ which we need to be invertible, which happens if and only if $\det X^TX \neq 0$. We have $$ \det X^TX = n\x^T\x - n^2\bar x^2 = n(\x^T\x - n\bar x^2) $$ and it's well-known that $$ \x^T\x - n\bar x^2 = \sum_{i=1}^n (x_i - \bar x)^2 $$ so $$ \det X^TX > 0\iff\sum_{i=1}^n (x_i - \bar x)^2 > 0. $$ If $\x \in \text{span}(\one)$ then $\det X^TX = 0$, and this also shows that the smaller the sample variance of $\x$ is, the more ill-conditioned this regression is.

Furthermore, under the assumption that $\text E(\e) = \mathbf0 $ and $\text E(\e\e^T) = \sigma^2 I$ we know $$ \text{Var}(\hat\beta) = \sigma^2 (X^TX)^{-1} $$ and $X^TX$ is 2x2 so we can work out that $$ (X^TX)^{-1} = \frac{1}{\det X^TX} \left[\begin{array}{cc}\x^T\x & -n\bar x \\ -n\bar x & n\end{array}\right] $$ which shows that the standard errors will also increase as $\x$ gets closer and closer to being in $\text{span}(\one)$, as measured by the sample variance of $\x$. This is just a consequence of the usual result that multicollinearity inflates standard errors but it's nice that we can so explicitly see how it does it here.

  • $\begingroup$ Thank you so much for the answer! I understand if there's multicollinearity in simple regression model, it means Xi is just a constant. But I'm not sure this logic can also be applied to the 'near' multicollinearity case. Is there any chance that Xi still has variation when 'near' multicollinearity exists between a and Xi? $\endgroup$
    – Dayeon
    Jun 13, 2019 at 6:46
  • $\begingroup$ @Dayeon yeah $X$ has perfect multicollinearity if and only if $\mathbf x$ has no variation. If there is any variation in $\mathbf x$ then there isn’t perfect multicollinearity and $X^TX$ is invertible. The more variation there is in $\mathbf x$ the less multicollinearity there is and the standard errors reflect that $\endgroup$
    – jld
    Jun 13, 2019 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.