6
$\begingroup$

What does it mean when statisticians talk about having more predictors than observations in a regression model? How could that even be possible? Why is it a problem in regression? Apologies, I am new to quant analysis and stats so not quite sure why this is the case? I would appreciate the simplest possible explanation -

$\endgroup$
  • 2
    $\begingroup$ Consider a dataset consisting of 100 images, each of which is 256 x 256. $\endgroup$ – Jakub Bartczuk Mar 18 '18 at 18:19
  • $\begingroup$ For a similar example see Olivetti faces dataset $\endgroup$ – Jakub Bartczuk Mar 18 '18 at 18:20
  • $\begingroup$ Sorry, this example isn't very clear to me but thank you $\endgroup$ – user3424836 Mar 18 '18 at 18:24
  • 4
    $\begingroup$ For a simple example, imagine if you had 5 students, and you wanted to predict their height from other variables. So you measure their sex, town, number of letters in their last name, shoe size, hair length, and weight. If you put these all in one model, you would have six predictors and only five observations. $\endgroup$ – Sal Mangiafico Mar 18 '18 at 19:15
  • $\begingroup$ Thank you, this is very helpful. Your answer made it clear to me what the problem is. $\endgroup$ – user3424836 Mar 18 '18 at 22:38
3
$\begingroup$

I think that the confusion comes from the way the word "observation" is sometimes used. Say that you wanted to know how the expression of 20,000 genes was related to some continuous biological variable like blood pressure. You have data both on expression of 20,000 genes and on blood pressure for 10,000 individuals. You might think that this involves 10,000 * 20,001 = 200,010,000 observations. There certainly are that many individual data points. But when people say there are "more predictors than observations" in this case, they only count each individual person as an "observation"; an "observation" is then a vector of all data points collected on a single individual. It might be less confusing to say "cases" rather than "observations," but usage in practice often has hidden assumptions like this.

The problem with more predictors than cases (usually indicated as "$p>n$") is that there is then no unique solution to a standard linear regression problem. If rows of the matrix of data points represent cases and columns represent predictors, there are necessarily linear dependences among the columns of the matrix. So once you've found coefficients for $n$ of the predictors, the coefficients for the other $(p-n)$ predictors can be expressed as arbitrary linear combinations of those first $n$ predictors. Other approaches like LASSO or ridge regression, or a variety of other machine-learning approaches, provide ways to proceed in such cases.

$\endgroup$
  • $\begingroup$ Thank you, this is very helpful. From what I gather based on your very comprehensive response, the problem with this kind of situation is that predictors would be correlated or collinear. Is my understanding of your explanation correct? $\endgroup$ – user3424836 Mar 18 '18 at 22:37
  • $\begingroup$ Any solution that minimizes the sum of squares will give a perfect fit. I think the overfitting problem is far more serious problem when the number of parameters (coefficients for predictor variables) exceeds the number of data points than the non-uniqueness of the solution. Note also when the number of parameters equals the number of data points there is a unique solution and you still have a perfect fit. $\endgroup$ – Michael Chernick Mar 18 '18 at 23:54
  • $\begingroup$ The techniques like LASSO that @EdM mentions are all variable selection techniques that reduce the number of predictor variables. Most of them provide ways of deciding which predictor variables are most important. $\endgroup$ – Michael Chernick Mar 19 '18 at 0:00
  • $\begingroup$ This is very helpful. When you say non-uniqueness of the solution, does that refer to the dataset? $\endgroup$ – user3424836 Mar 19 '18 at 0:05
  • $\begingroup$ @user3424836 it has to do with the general structure of the data, not the further details of the data set. Any situation with $p>n$ will have this problem, whether you think about it as non-unique linear-regression solutions like I described or overfitting like Michael Chernick describes. $\endgroup$ – EdM Mar 19 '18 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.