What does it mean when statisticians talk about having more predictors than observations in a regression model? How could that even be possible? Why is it a problem in regression? Apologies, I am new to quant analysis and stats so not quite sure why this is the case? I would appreciate the simplest possible explanation -
I think that the confusion comes from the way the word "observation" is sometimes used. Say that you wanted to know how the expression of 20,000 genes was related to some continuous biological variable like blood pressure. You have data both on expression of 20,000 genes and on blood pressure for 10,000 individuals. You might think that this involves 10,000 * 20,001 = 200,010,000 observations. There certainly are that many individual data points. But when people say there are "more predictors than observations" in this case, they only count each individual person as an "observation"; an "observation" is then a vector of all data points collected on a single individual. It might be less confusing to say "cases" rather than "observations," but usage in practice often has hidden assumptions like this.
The problem with more predictors than cases (usually indicated as "$p>n$") is that there is then no unique solution to a standard linear regression problem. If rows of the matrix of data points represent cases and columns represent predictors, there are necessarily linear dependences among the columns of the matrix. So once you've found coefficients for $n$ of the predictors, the coefficients for the other $(p-n)$ predictors can be expressed as arbitrary linear combinations of those first $n$ predictors. Other approaches like LASSO or ridge regression, or a variety of other machine-learning approaches, provide ways to proceed in such cases.