After reading material related to my topic, I understood that multicollinearity among predictors would result in singular matrix $X'X$, and that leads to noninvertible matrix. Thus, the solution will not be unique.

Now, I am confused after reading that having too many variables (number of predictors greater than number of observations) causes the matrix $X'X$ to be singular too.

Is that true in both situations? If yes, could you explain that, please?

  • 1
    $\begingroup$ "Not acceptable" is quite an exaggeration. Concerning your question, see these related threads. $\endgroup$ – whuber Jul 15 '16 at 17:52
  • 1
    $\begingroup$ Adding a regularization will make it work. see lasso regression and ridge regression $\endgroup$ – Haitao Du Jul 15 '16 at 17:53
  • $\begingroup$ Do you have any remaining questions or concerns after reading see.stanford.edu/materials/lsoeldsee263/08-min-norm.pdf ? $\endgroup$ – Mark L. Stone Jul 15 '16 at 17:58
  • $\begingroup$ What exactly do you not understand about why those are, in essence, the same problems? It is a math fact that if $p > n$, then any column of $X$ can be expressed as a linear combination of the other columns. That's multicollinearity. $\endgroup$ – AdamO Jul 15 '16 at 18:21
  • $\begingroup$ @ AdamO, so that means multicollinearity = too many variables $\endgroup$ – jeza Jul 15 '16 at 18:31

Suppose you add predictors to your model one by one, and you have a way to do that to minimize the multicollinearity issue - that is, you somehow make sure that each new predictor is independent of the predictors that are already in the model. That way, the correlation matrix for predictors will be always diagonal, and multicollinearity won't be an issue - up to a point.

The thing is that, for a given sample size, $N$, it's impossible to find more than $N$ independent predictors (including the column of 1s for the intercept). That's because for design matrix $X$ of size $N \times p$ its rank cannot be greater than $\text{min}\{N, p\}$. No matter how you pick the predictors, if you use more than $N$ of them, the $X'X$ will be non-invertible for sure.

Based on what we know from linear algebra, it's impossible for $X'X$ to have rank greater than that of $X$. So, with $p > N$ then $X'X_{p\times p}$ will be of rank $N$ at most, but in order for it to be invertible the rank has to be $p$.

  • $\begingroup$ "if you use more than $N$ of them, the $X'X$ will be non-invertible for sure." , could please clarify this a little bit further ? $\endgroup$ – jeza Jul 15 '16 at 18:47
  • $\begingroup$ @ Richard Hardy , Are "Too many variables and multicollinearity " have the same idea in OLS regression. $\endgroup$ – jeza Jul 15 '16 at 20:06
  • $\begingroup$ @jeza: "Too many variables" (e.g. more than $N$) guarantees multicollinearity, for the linear algebra reasons Nik Tuzov gives. But you can face multicollinearity in some situations with very few variables too. $\endgroup$ – Henry Jul 15 '16 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.