# Find minimal set of variables for regression (PCA?)

Given a large matrix with 10,000 rows (variables) and 20 columns (sampling timepoints), I am trying to build a linear model to find the sampling timepoint given some of the variables. So in essence, I want to find a small subset of variables that explains the data well enough that I can build a linear model like timepoint ~ v1 + v2 + v3 + v4. I hope that this will allow me to only take measurements of the few variables that actually make a difference instead of measuring all 10,000 of them every time.

I have already tried using PCA for this, by doing a PCA on the whole matrix and then using the PCs as my variables. This works to some degree, but it does not actually solve my problem, since I would still have to measure all the variables for this to work. When I look at the loadings for the different PCs I usually get a large number of variables for each of them, so I cannot use this to determine the important variables either.

Is there a way I can do this?

• Welcome to the site. One note: Usually observations are rows and variables are columns - just something to be aware of when you write a program to do whatever you wind up doing. One problem I can see is that, with so many more variables than observations, there will doubtless be many sets of variables that perfectly identify the subjects. Even your example linear model has 4 variables, which is probably overfit. Can you get more timepoints? If not, maybe look at each variable individually. Mar 7, 2013 at 11:48

I general I would say look at something like LASSO (which is illustrated nicely in the answers to this similar question: Detecting significant predictors out of many independent variables).

However, in you case you only have 20 sample points, so even LASSO is going to struggle to do anything useful.

If you reall can't get many more datapoints then you may have to accept that (until more data comes in) you will have a pretty poor model. Accepting that I would probably proceed by testing each of the 10,000 predictors in turn to get a list of 10,000 univariate p-values, and then use False Discover Rate control to select a small set of the variables such that most of them are probably useful. This will of course potentially leave you with repetitions (i.e. if two variables are very similar both would be incldued), but at super low sample sizes you will always have a compromise.

This should really be a comment to Corone's answer, but I don't have reputation to comment yet...

What about a greedy type search? When you are done with the 10k univariate test, you can pick the most powerful one, say it's v100. Include it in your model and test for all k != 100

timepoint ~ v100 + vk

keep adding variables until you are satisfied.

• Other people have had this idea, too jf, so you're in good company: after some improvement, it was called "[forward] stepwise regression." For some indications of the problems with this, please search our site for that phrase. With 10K variables, the problems will be so overwhelming that there is little hope for this approach.
– whuber
Mar 8, 2013 at 22:16
• @whuber while I agree with your conclusion, I think people are often too quick to dismiss "stepwise methods" as bad. There is nothing wrong with "stepwise", the problem is the repeated application of a test for variables significance. What jf328 hasn't addressed is how you decide whether the model is better with or without the new variable. If that criteria is well defined, then stepwise simply means gradient descent, and where would we be without that! However, I suspect in this case that ANY well defined metric will simply refuse all but one predictor. Mar 8, 2013 at 22:52
• @Corone In this situation, we would be much better off forgetting about gradient descent: it is virtually guaranteed to find local minima that are not global.
– whuber
Mar 9, 2013 at 4:15
• @whuber, I totally agree. My comment was merely that isn't the "stepwise" part that kills us, it's the inability to define a sensible surface to minimise. Possibly heading OT though, so will shut up. Mar 9, 2013 at 6:32