# Choosing model for more predictors than observations

I'm working with a data consisting of 1000 observations of 2000 predictors and one variable we wish to predict. There are couple of problems I can't get around. I am aware that such setting has been adressed in some previous questions, however, I'm not exactly sure how to put all those pieces together.

My goal is to choose model with proper features. First of all, to deal with $$p>n$$ problem, I wanted to run Lasso (i.e. Lasso function included in HDCI package in R). However, I'm not sure how to set proper $$\lambda$$ value for my data and how to interprete the results. Am I right in saying that checking for $$\beta > x$$, with $$x$$ being some properly chosen value, would filter out irrelevant variables?

The other thing is that some of 2000 predictors might be collinear and as such would "survive" Lasso in favor of other, perhaps significant predictors. What is the best way to seek for such multicollinearity before running any model? That is: how to efficiently test which of 2000 columns, each being of 1000 length, are actually collinear? I tried running VIF on my data frame, but well, it's definitely way too complex.

The main purpose of constructing this model is to use it later to make prediction on the set of 500 observations, however without the variable we are trying to predict.

• is this time series data or is it cross-sectional ? – IrishStat May 9 '19 at 20:17
• @IrishStat It's cross-sectional - we are trying to predict level of some particular protein, based on other factors measured across 1000 patients. – Wham Bang Shang-a-Lang May 9 '19 at 20:21
• ok because if it was time series one would have to be concerned with contemporaneous and possible lag effects for each of the candidate predictors. – IrishStat May 9 '19 at 20:24
• Please say a bit more about the purpose of your modeling and how you wish to apply your results. Is your primary interest in developing a model for performing predictions on new cases, or something else? If it's for performing predictions, will you always have all 2000 predictors available or are you interested in cutting that down to a smaller number for use in predicting new cases? – EdM May 9 '19 at 20:59
• @EdM Main purpose is to train a model which I could use to predict value of that protein provided another data frame of observations of those 2000 variables, but without the level of protein we wish to predict. We have 2000 predictors, however some of them are just noise, some are multicollinear - the purpose is to filter them out and select only those which are actually important for us. – Wham Bang Shang-a-Lang May 10 '19 at 9:23

## 2 Answers

Ridge regression has the nice property of squashing away collinearity.

Why does Ridge Regression work well in the presence of multicollinearity?

LASSO has the nice property of eliminating variables, as you've noted.

The combination of the two is called an elastic net, and it both squashes away collinearity and does variable reduction.

The way to tune your elastic net is via cross validation. This means that you divide your 1000 observations into groups of 200 each: G1, G2, G3, G4, and G5 (or ten sets of 100, or 1000 sets of one). Then you train your model on four and see how it does on the fifth. Repeat this until you have tested on all five. Keep track of the performance, because you're about to do it again for new elastic net parameters, then do it again and again as you figure out a good elastic net to use. (This approach is valid if you're just doing ridge or LASSO, too.)

I found this video very helpful: https://www.youtube.com/watch?v=ctmNq7FgbvI. Even if you will be doing your regression in another language (Python or SAS, perhaps), the ideas are valid, and you'll adjust the syntax for the language you use.

• Thanks a lot for your comment! Because of it I started digging into Lasso and Ridge regression more and it seems to be just what I was looking for. The YouTube channel you recommended is of great help in doing so :) – Wham Bang Shang-a-Lang May 11 '19 at 14:56

The answer from @Dave gets to the main points: use ridge regression and/or LASSO to develop your model, and use cross validation to select the level of penalization (often called $$\lambda$$ in LASSO or ridge regression ). Chapter 6 of An Introduction to Statistical Learning has worked examples of ridge and LASSO, showing how to use the built-in cross-validation tools of the glmnet package in R to select penalization levels that optimize the prediction of your outcome variable. Here are some more details than could fit in a comment on his answer .

In general, you don't want to throw away useful information if you will be using your model to predict new cases for which you have the predictors but not the outcome values. Your concern that LASSO might select one from a set of collinear predictors and thus omit other important correlated predictors is valid, although in practice what happens is that the selected predictor serves as a proxy for the predictors that were omitted. If that bothers you then ridge regression has the advantage of not throwing away information from any predictors, instead weighting them according to their relations to the outcome variable.

A few thoughts on implementation. First, as many of your predictors might simply be noise, you could consider omitting predictors that do not vary substantially among cases or that are at such low levels that they might not be reliable for your prediction work. That's often done in analysis of microarray or RNA seq data. It's best to do that initial variable removal without looking at the relations to the outcome.

Second, you have to be careful that the remaining predictors are pre-scaled similarly. The penalization is applied to the magnitudes (LASSO) or squares (ridge) of the regression coefficients, so to choose the weightings among them fairly they all have to be on similar scales. Continuous predictors are typically adjusted to zero mean and unit variance before the analysis. Some programs do this automatically and then re-convert the coefficients to the original scales. Other software might require you to do that work. Be sure you know how your software deals with that issue.

Finally, as you want to use ridge and LASSO in the context of a linear regression, you also need to verify the usual requirements of linear regression. For example, are your predictors related linearly to the outcome variable in their original scales, or is some transformation necessary to obtain linearity? For example, mRNA expression data often will work better in this respect if they are first log-transformed. You also might need to consider some transformation of the outcome variable.