Consequence of Multicollinearity

In case of perfect multicollinearity the predictor matrix is singular and therefore cannot be inverted . Under these circumstances, the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ does not exist (Wikipedia) .

I can't visualize the situation.

• When does the situation of perfect multicollinearity occur ?

• In case of perfect multicollinearity, why is the predictor matrix singular ?

• Under these circumstances, why does the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ not exist ?

• You may find my answer here helpful for your third bullet point. – TrynnaDoStat Mar 29 '15 at 15:07

• The situation of perfect multicollinearity occurs when one (or more) column(s) of the design matrix are linear combinations of the others, i.e. the set of column vectors of the matrix $\mathbf{X}$ is linearly dependent.
• As you might recall from your linear algebra course, if a matrix has size $n \times p$, with $p < n$, the maximum rank of $\mathbf{X}$ will be $p$. As the columns are not independent the rank will be lower than $p$. However, to be formal, the terms singular / non singular refer only to square matrices. In this case you should say that $\mathbf{X}$ is not full column rank.
• You can see that you need to invert the matrix $\mathbf{X}'\mathbf{X} \in \mathbb{R}^{p\times p}$, for which the property $\mathrm{rk}(\mathbf{X}) = \mathrm{rk}(\mathbf{X}'\mathbf{X})$ holds. It is clear that it can't be inverted. In fact $\mathbf{X}'\mathbf{X}$ will be nonsingular only if its rank equals $p$, but as $\mathrm{rk}(\mathbf{X}) < p$ it will be singular. In the case of the matrix $\mathbf{X}$ having rank $p < n$, i.e. having dependent columns, the system of normal equations $\mathbf{X}'\mathbf{X} \boldsymbol{\beta} = \mathbf{X}'\mathbf{y}$ will have infinite solutions depending on $n-p$ variables. There are infinitely many vectors that solve our initial minimisation problem: $$\min_{\boldsymbol{\beta}} \| \mathbf{y} - \mathbf{X}\boldsymbol{\beta} \|.$$ You can choose, for example, the one having minimum norm.
The formula using $(X'X)^{-1}$ isn't applicable because the inverse matrix doesn't exist. However, any $\beta$ that satisfies the normal equations $X'X\beta=X'y$ (there will be infinitely many of these solutions) is a least squares solution.
• If i get many $\hat\beta$ in such circumstance, do all of them unbiased estimate of $\beta$ ? – ABC Mar 29 '15 at 17:14
• You have to start by specifying which of the least squares solutions you want to consider. For example, you might pick the solution that minimizes $\| \hat{\beta} \|$ among all of the least squares solutions. Unfortunately, this isn't an unbiased estimator. – Brian Borchers Mar 29 '15 at 17:21