# Fitting least squares when number of predictors are larger than instances

A statement from the book Introduction to Statistical learning with applications in R, didn't quite make sense to me. It says, "In cases when number of predictors are greater than the instances we cannot even fit the multiple linear regression model using least squares". Why is that?

• This leads to multicollinearity. I have not read that book but I guess they mean $X^TX$ cannot be inverted, so the formula $\beta=(X^TX)^{-1}X^Ty$ cannot be used. Commented Jan 18, 2017 at 3:30
• You can pick the minimum norm least squares solution to break the infinite-way tiie for least squares solution. Read this see.stanford.edu/materials/lsoeldsee263/08-min-norm.pdf . As for @gung s answer, identifiability, schmidenitifability. If we're trying to make predictions, rather than trying to estimate coefficients, we dom't need no stinkim' identifiability. Commented Jan 18, 2017 at 4:59
• The quoted text says that F statistic can't be used when the number er of predictors are greater than the instances. But I think it makes too strong a statement, or at least a statement open to misinterpretation, in saying that a least squares model can not be fit when number of predictors is greater than number of instances (data points). Commented Jan 18, 2017 at 5:11
• @MarkL.Stone gung's answer is very informative at the level of the OP, but I truly would appreciate your giving a formal answer discussing how this is modified when regularization comes into play, for instance. I also think further discussion on matrix invertability and null spaces could be enlightening (this at a more basic level than your comments). Best wishes! Commented Jan 18, 2017 at 13:20
• @Antoni Parellada I'll let you have the honor/get the glory and points from providing the answer. Thanks for the suggestion though. Commented Jan 18, 2017 at 13:32

$$5 = x_1 + x_2$$ What numbers do you think $x_1$ and $x_2$ might be? They really could be anything. Here are some possibilities: