# What is the problem with $p > n$?

I know that this is the solving system of linear equation problem.

But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen?

Does not the data collection come from the delicate survey design or experimental design to the extent that they at least think about this thing?

If the data collection want to collect 45 variables to conduct research then why would he collect less than 45 observation? Did I miss something and although the model selection part also eliminated the non-improvement variables on the response and always the collected variable will be eliminated to $$45-(45-p)$$ right?

So then why would we face the non-unique solution in those case?

• I suspect you have not yet been enlightened about step-wise model selection, but I like this question. Jan 5 '19 at 4:53
• Yes, no doubt I don't. I get the concept part but partly about the combination of the model in math. why would we suffer from p > n? Jan 5 '19 at 5:00
• There might be cases where, once you got a subject in your sample, it is cheap to measure (tens of) thousands of variables---at least not more expensive than 10 or 100. Bioinformatics? automated sensors? Jan 5 '19 at 10:14
• Sometimes it just is not feasible to have enough observations to counter all the variables that need to be accounted for. Classic examples are medical studies, where the number of people you can treat is very limited.
– nope
Jan 5 '19 at 11:43

This could occur in many scenarios, few examples are:

1. Medical data analysis at hospitals. Medical researchers studying a particular cancer primarily can do data collection at their own hospital, and I think it is not a bad thing that they try collect many variables as possible from one particular patient like age, gender, tumour size, MRI, CT volume.
2. Micro platereader array studies in bioinformatics. It is often the case that you don’t have many species but you want to be able to test for as many effects as possible.
3. Analysis with images. You have often 16 million pixels while it is very difficult to collect and store that many images.
4. MRI reconstructions are often similar problems, which need sparse regression techniques, and improving them is really a central question in MRI imaging research.

The solution is really, to look at the regression literature and find what best works for your application.

1. If you have domain knowledge, incorporate into your prior distribution and take a Bayesian approach with Bayesian Linear Regression.

2. If you want to find a sparse solution, automatic relevance determination’s empirical Bayes approach could be the way to go.

3. If you think that with your problem, having a notion of probabilities is inappropriate (like solving a linear systems of equations), it might be worth to look at the Moore-Penrose pseudoinverse.

4. You can approach it from a feature selection perspective, and reduce the number of p until it is a well-posed problem.

• Thank you a lot for the answer, I would be grateful if you give me a case for study. From what I heard from you it sounds like "the case where it is rare and the researcher really want to dig deep into that thing with the huge amount of variable they think they will extract the useful pattern and information to predict that rarity?" Jan 5 '19 at 13:53
• You can call it a rare cancer disease, but in my experience it takes time to gather enough data even for more common ones if you are limited to one hospital’s patients. The key is that n is scarce, while p is not. I’m still looking for a good, practical example, but David Wipf’s work on neuroimaging by overcomplete dictionaries might be a good start from the applied theoretical perspective. Jan 6 '19 at 21:56

This is a very good question. When the number of candidate predictors $$p$$ is more than the effective sample size $$n$$, and one does not place any restrictions on the regression coefficients (e.g., one is not using shrinkage, a.k.a. penalized maximum likelihood estimation or regularization), the situation is hopeless. I say that for several reasons including

• If you think about the number of non-redundant linear combination of variables that can be analyzed, this number is $$\leq \min(n, p)$$. For example you can't even compute, much less trust, principal components beyond $$\min(n, p)$$.
• With $$p = n$$ and no two $$y$$-coordinates on a vertical line when plotting $$(x, y)$$, one can achieve $$R^{2}=1.0$$ for any dataset even if the true population $$R^2$$ is 0.0.
• If you use any feature selection algorithm such as dreaded stepwise regression models, the list of features "selected" will essentially be a random set of features with no hope of replicating in another sample. This is especially true if there are correlations among the candidate features, e.g., co-linearity.
• The value of $$n$$ needed to estimate with decent precision a single correlation coefficient between two variables is about 400. See here.

In general, a study that intends to analyze 45 variables on 45 subjects is poorly planned, and the only ways to rescue it that I know of are

• Pre-specify one or two predictors to analyze and ignore the rest
• Use penalized estimation such as ridge regression to fit all the variables but take the coefficients with a grain of salt (heavy discounting)
• Use data reduction, e.g., principal components, variable clustering, sparse principal components (my favorite) as discussed in my RMS book and course notes. This involves combining variables that are hard to separate, and not trying to estimate separate effects for them. For $$n=45$$ you may only get by with 2 collapsed scores for playing against $$y$$. Data reduction (unsupervised learning) is more interpretable than most other methods.

A technical detail: if you use one of the best combination variable selection/penalization methods such as lasso or elastic net you can lower the chance of overfitting but will ultimately be disappointed that the list of selected features is highly unstable and will not replicate in other datasets.

• This is really helpful Frank! I get the knowledge of how to deal with the cases where p > n, I get to know why it is a problem and what kind of problem can lead to. I really got a lot of answer for that a lot from you, although you miss my point, I only ask a simple question that why that case might happen if they make a delicate data collection. Statisticians know it can be a problem so why they did not prevent it, obviously they know, that's mean they try but cannot. so why they cannot? Thank you so much !!! Jan 5 '19 at 13:52
• That question is more about psychology and logistics. I find that very often studies are designed by committee and everyone has a favorite variable. Before you realize it the list of variables is too long to be able to (1) reliably measure all of them and (2) analyze them. Jan 5 '19 at 15:47