Find minimal set of variables for regression (PCA?) Given a large matrix with 10,000 rows (variables) and 20 columns (sampling timepoints), I am trying to build a linear model to find the sampling timepoint given some of the variables. So in essence, I want to find a small subset of variables that explains the data well enough that I can build a linear model like timepoint ~ v1 + v2 + v3 + v4. I hope that this will allow me to only take measurements of the few variables that actually make a difference instead of measuring all 10,000 of them every time.
I have already tried using PCA for this, by doing a PCA on the whole matrix and then using the PCs as my variables. This works to some degree, but it does not actually solve my problem, since I would still have to measure all the variables for this to work. When I look at the loadings for the different PCs I usually get a large number of variables for each of them, so I cannot use this to determine the important variables either.
Is there a way I can do this?
 A: I general I would say look at something like LASSO (which is illustrated nicely in the answers to this similar question: Detecting significant predictors out of many independent variables).
However, in you case you only have 20 sample points, so even LASSO is going to struggle to do anything useful.
If you reall can't get many more datapoints then you may have to accept that (until more data comes in) you will have a pretty poor model.  Accepting that I would probably proceed by testing each of the 10,000 predictors in turn to get a list of 10,000 univariate p-values, and then use False Discover Rate control to select a small set of the variables such that most of them are probably useful.  This will of course potentially leave you with repetitions (i.e. if two variables are very similar both would be incldued), but at super low sample sizes you will always have a compromise.
A: This should really be a comment to Corone's answer, but I don't have reputation to comment yet...
What about a greedy type search? When you are done with the 10k univariate test, you can pick the most powerful one, say it's v100. Include it in your model and test for all k != 100
timepoint ~ v100 + vk
keep adding variables until you are satisfied.
