0
$\begingroup$

I am working with a data set that has 18500 dimensions and 450 observations(p>>n). I am coding on R and I am looking at using LASSO, Ridge and Elastic net regression. I am also looking into performing some form of variable selection with PCA or SVD. I know that I want to compare different models, first using Ridge, LASSO and Elastic Net on the full data set. I am considering comparing these models to a model fitted after performing PCA or SVD. In what manner should I compare my models considering the fact that I am attempting to perform a prediction task? I am considering comparing MSE values as well as the correlation between y and y_hat values.

I was also wondering if anybody could recommend a place where I could find some useful code and packages in order to undertake this task in R? Any help would be much appreciated. Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

SVD- or PCA-based approaches to regression do not perform selection among your p = 18500 predictor variables. They can perform dimension reduction, lowering the dimension of the predictor space to less than p. Each of the retained dimensions, however, could be a linear combination of all 18500 of your original predictors. So you don't necessarily select any of the original predictors, just re-weight them.

Elastic net and LASSO do perform variable selection, although as with any variable-selection system there is no assurance that the particular variables selected would be the same from experiment to experiment.

A useful way to proceed, long used in analysis of microarray data, can be to remove predictors whose values show low variance among observations; that's similar to what you probably intend with PCA/SVD but is easier to explain to others (not inconsequential if your target audience is biologists). Those would be housekeeping and other genes that are unlikely to be related to your outcome of interest.

If you are really interested in prediction and will have information on the same genes for the cases you wish to predict, then there is probably little to gain by further selection of predictors. Ridge regression will do appropriate penalization, and it bears a very strong relation to principal-components regression as explained on page 79 of ESLII, simply weighting all of the principal components rather than tossing some out completely. It thus should pretty much accomplish what you are intending to do with prior SVD/PCA. ISLR has nicely worked-through examples of ridge regression and LASSO in R, starting on page 251; the generalization to elastic net is straightforward if perhaps more computationally expensive.

For testing your models, you need to consider carefully what types of prediction errors matter to you and design the tests appropriately. If you primarily care about MSE then that would be fine, but that might not be the best indicator of what you really care about for future use of the model. Repeating the model-building process on multiple bootstrap samples of the original data and testing them for prediction ability on the original data is one good way to proceed. The rms package in R has many useful functions to aid in building and testing regression models, and is well worth the somewhat steep learning curve if you will be doing a lot of this type of work.

With your underlying data coming from microarrays, however, you also have to be aware of the peculiarities of such data, with batch effects complicating comparison of results from experiment to experiment, and significant pre-processing often necessary even to ensure within-experiment consistency. Bioconductor has well-developed tools, like the limma package, to handle such data and to help identify genes that are most closely related to your outcome of interest. If you are not already familiar with those tools and particularly if your microarray data have not already been appropriately pre-processed, look into those tools carefully.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.