1
$\begingroup$

In case of perfect multicollinearity the predictor matrix is singular and therefore cannot be inverted . Under these circumstances, the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ does not exist (Wikipedia) .

I can't visualize the situation.

  • When does the situation of perfect multicollinearity occur ?

  • In case of perfect multicollinearity, why is the predictor matrix singular ?

  • Under these circumstances, why does the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ not exist ?

$\endgroup$
  • $\begingroup$ You may find my answer here helpful for your third bullet point. $\endgroup$ – TrynnaDoStat Mar 29 '15 at 15:07
8
$\begingroup$
  • The situation of perfect multicollinearity occurs when one (or more) column(s) of the design matrix are linear combinations of the others, i.e. the set of column vectors of the matrix $\mathbf{X}$ is linearly dependent.
  • As you might recall from your linear algebra course, if a matrix has size $n \times p$, with $p < n$, the maximum rank of $\mathbf{X}$ will be $p$. As the columns are not independent the rank will be lower than $p$. However, to be formal, the terms singular / non singular refer only to square matrices. In this case you should say that $\mathbf{X}$ is not full column rank.
  • You can see that you need to invert the matrix $\mathbf{X}'\mathbf{X} \in \mathbb{R}^{p\times p}$, for which the property $\mathrm{rk}(\mathbf{X}) = \mathrm{rk}(\mathbf{X}'\mathbf{X})$ holds. It is clear that it can't be inverted. In fact $\mathbf{X}'\mathbf{X}$ will be nonsingular only if its rank equals $p$, but as $\mathrm{rk}(\mathbf{X}) < p$ it will be singular. In the case of the matrix $\mathbf{X}$ having rank $p < n$, i.e. having dependent columns, the system of normal equations $\mathbf{X}'\mathbf{X} \boldsymbol{\beta} = \mathbf{X}'\mathbf{y}$ will have infinite solutions depending on $n-p$ variables. There are infinitely many vectors that solve our initial minimisation problem: $$ \min_{\boldsymbol{\beta}} \| \mathbf{y} - \mathbf{X}\boldsymbol{\beta} \|.$$ You can choose, for example, the one having minimum norm.
$\endgroup$
9
$\begingroup$

It might be helpful to add that in this situation there will be infinitely many least squares solutions. The problem is not that a least squares solution does not exist, but rather that the least squares solution is not unique.

The formula using $(X'X)^{-1}$ isn't applicable because the inverse matrix doesn't exist. However, any $\beta$ that satisfies the normal equations $X'X\beta=X'y$ (there will be infinitely many of these solutions) is a least squares solution.

$\endgroup$
  • $\begingroup$ If i get many $\hat\beta$ in such circumstance, do all of them unbiased estimate of $\beta$ ? $\endgroup$ – ABC Mar 29 '15 at 17:14
  • 2
    $\begingroup$ You have to start by specifying which of the least squares solutions you want to consider. For example, you might pick the solution that minimizes $\| \hat{\beta} \|$ among all of the least squares solutions. Unfortunately, this isn't an unbiased estimator. $\endgroup$ – Brian Borchers Mar 29 '15 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.