Choosing model for more predictors than observations

Question

I'm working with a data consisting of 1000 observations of 2000 predictors and one variable we wish to predict. There are couple of problems I can't get around. I am aware that such setting has been adressed in some previous questions, however, I'm not exactly sure how to put all those pieces together.

My goal is to choose model with proper features. First of all, to deal with $p>n$ problem, I wanted to run Lasso (i.e. Lasso function included in HDCI package in R). However, I'm not sure how to set proper $\lambda$ value for my data and how to interprete the results. Am I right in saying that checking for $\beta > x$, with $x$ being some properly chosen value, would filter out irrelevant variables?

The other thing is that some of 2000 predictors might be collinear and as such would "survive" Lasso in favor of other, perhaps significant predictors. What is the best way to seek for such multicollinearity before running any model? That is: how to efficiently test which of 2000 columns, each being of 1000 length, are actually collinear? I tried running VIF on my data frame, but well, it's definitely way too complex.

The main purpose of constructing this model is to use it later to make prediction on the set of 500 observations, however without the variable we are trying to predict.

Answer 1

Ridge regression has the nice property of squashing away collinearity.

Why does Ridge Regression work well in the presence of multicollinearity?

LASSO has the nice property of eliminating variables, as you've noted.

The combination of the two is called an elastic net, and it both squashes away collinearity and does variable reduction.

The way to tune your elastic net is via cross validation. This means that you divide your 1000 observations into groups of 200 each: G1, G2, G3, G4, and G5 (or ten sets of 100, or 1000 sets of one). Then you train your model on four and see how it does on the fifth. Repeat this until you have tested on all five. Keep track of the performance, because you're about to do it again for new elastic net parameters, then do it again and again as you figure out a good elastic net to use. (This approach is valid if you're just doing ridge or LASSO, too.)

I found this video very helpful: https://www.youtube.com/watch?v=ctmNq7FgbvI. Even if you will be doing your regression in another language (Python or SAS, perhaps), the ideas are valid, and you'll adjust the syntax for the language you use.

Answer 2

The answer from @Dave gets to the main points: use ridge regression and/or LASSO to develop your model, and use cross validation to select the level of penalization (often called $\lambda$ in LASSO or ridge regression ). Chapter 6 of An Introduction to Statistical Learning has worked examples of ridge and LASSO, showing how to use the built-in cross-validation tools of the glmnet package in R to select penalization levels that optimize the prediction of your outcome variable. Here are some more details than could fit in a comment on his answer .

In general, you don't want to throw away useful information if you will be using your model to predict new cases for which you have the predictors but not the outcome values. Your concern that LASSO might select one from a set of collinear predictors and thus omit other important correlated predictors is valid, although in practice what happens is that the selected predictor serves as a proxy for the predictors that were omitted. If that bothers you then ridge regression has the advantage of not throwing away information from any predictors, instead weighting them according to their relations to the outcome variable.

A few thoughts on implementation. First, as many of your predictors might simply be noise, you could consider omitting predictors that do not vary substantially among cases or that are at such low levels that they might not be reliable for your prediction work. That's often done in analysis of microarray or RNA seq data. It's best to do that initial variable removal without looking at the relations to the outcome.

Second, you have to be careful that the remaining predictors are pre-scaled similarly. The penalization is applied to the magnitudes (LASSO) or squares (ridge) of the regression coefficients, so to choose the weightings among them fairly they all have to be on similar scales. Continuous predictors are typically adjusted to zero mean and unit variance before the analysis. Some programs do this automatically and then re-convert the coefficients to the original scales. Other software might require you to do that work. Be sure you know how your software deals with that issue.

Finally, as you want to use ridge and LASSO in the context of a linear regression, you also need to verify the usual requirements of linear regression. For example, are your predictors related linearly to the outcome variable in their original scales, or is some transformation necessary to obtain linearity? For example, mRNA expression data often will work better in this respect if they are first log-transformed. You also might need to consider some transformation of the outcome variable.