# More predictors than observations?

What does it mean when statisticians talk about having more predictors than observations in a regression model? How could that even be possible? Why is it a problem in regression? Apologies, I am new to quant analysis and stats so not quite sure why this is the case? I would appreciate the simplest possible explanation -

• Consider a dataset consisting of 100 images, each of which is 256 x 256. – Jakub Bartczuk Mar 18 '18 at 18:19
• For a similar example see Olivetti faces dataset – Jakub Bartczuk Mar 18 '18 at 18:20
• Sorry, this example isn't very clear to me but thank you – user3424836 Mar 18 '18 at 18:24
• For a simple example, imagine if you had 5 students, and you wanted to predict their height from other variables. So you measure their sex, town, number of letters in their last name, shoe size, hair length, and weight. If you put these all in one model, you would have six predictors and only five observations. – Sal Mangiafico Mar 18 '18 at 19:15
• Thank you, this is very helpful. Your answer made it clear to me what the problem is. – user3424836 Mar 18 '18 at 22:38

The problem with more predictors than cases (usually indicated as "$p>n$") is that there is then no unique solution to a standard linear regression problem. If rows of the matrix of data points represent cases and columns represent predictors, there are necessarily linear dependences among the columns of the matrix. So once you've found coefficients for $n$ of the predictors, the coefficients for the other $(p-n)$ predictors can be expressed as arbitrary linear combinations of those first $n$ predictors. Other approaches like LASSO or ridge regression, or a variety of other machine-learning approaches, provide ways to proceed in such cases.
• @user3424836 it has to do with the general structure of the data, not the further details of the data set. Any situation with $p>n$ will have this problem, whether you think about it as non-unique linear-regression solutions like I described or overfitting like Michael Chernick describes. – EdM Mar 19 '18 at 1:51