How to identify chance in feature selection?

Question

I am working on bioinformatics data. I have a small dataset of around 60 rows. I have 3000 features. I need to build a regressor. I will be applying an initial round of feature selection to reduce the number of features to around 40. I am thinking of picking up the top 40 features which correlate well with the output variable (cross-validated feature selection the right way as suggested by https://www.nodalpoint.com/not-perform-feature-selection/ to avoid data leakage). Then I am planning to do an exhaustive feature selection of subsets of 3 or 4 features and input them to a Gaussian process regressor (to accomodate non-linear regressor). I will be doing feature selection and model evaluation using nested cross-validation. I hope I am taking all necessary precautions to build a model that can truly incorporate the relationship between my features and my response variable. However, because I have a huge initial feature set of 3000 features, I am afraid that I might end up picking some features that correlate well with my response variable just by luck or chance. And because I am doing exhaustive feature selection to select a smaller feature set, again chance might play a role.

What are the standard techniques to rule out the chance of luck in scenarios like this? Kindly share your inputs and links to any academic articles.

Many thanks for your time and attention!

Answer 1

You are in a situation where the number of potential predictors (3000) far exceeds the number of cases (60), so you can't just do ordinary linear regression. With only 60 cases you probably can only include 4 to 6 effective predictors in your final model to avoid overfitting (10-15 cases per predictor). You have to find a principled way to cut down the effective number of predictors, to allow modeling without overfitting.

There are well-developed tools for dealing with such situations. An Introduction to Statistical Learning is a good resource for starting out. Much of the development of those tools was motivated by this type of situation in gene-expression studies, with thousands of potential predictors but only a few dozens of cases.

If prediction on new cases is your primary interest, it's possible to include all predictors with ridge regression. You also could consider principal-component regression (PCR) or partial least squares regression (PLS). In practice with correlated predictors those latter two approaches might keep contributions from most or all of the original predictors. In all three approaches the effective dimension of the predictor set is reduced, with coefficient values lower in magnitude than what they would be in a full model. That makes the model more likely to apply adequately to new cases.

If selection among the predictors is important, you could use LASSO to choose a small number of predictors while penalizing coefficients to avoid overfitting. There's also a hybrid of ridge and LASSO, called elastic net.

All of those approaches require choices: the number of effective predictor dimensions to keep (PCR, PLS), or a penalty term to reduce the magnitudes of regression coefficients (ridge, LASSO) and, with LASSO, the number of retained original predictors. In these approaches, you typically use cross-validation to make those choices in a way that, say, minimizes cross-validated error.

But cross-validation for feature selection without penalization, as you seem to propose, is not a good idea. If you want to get an estimate of the reliability of your modeling process, repeat the entire process on multiple bootstrapped samples of the original data set and see how well those models explain your original data set.

With LASSO you will find that the retained predictors vary among the models based on the bootstrapped samples, but that's inevitable when you have a large number of correlated predictors. Despite that variability in the identities of the retained predictors, LASSO still can work well for predicting new cases.