I am working on bioinformatics data. I have a small dataset of around 60 rows. I have 3000 features. I need to build a regressor. I will be applying an initial round of feature selection to reduce the number of features to around 40. I am thinking of picking up the top 40 features which correlate well with the output variable (cross-validated feature selection the right way as suggested by https://www.nodalpoint.com/not-perform-feature-selection/ to avoid data leakage). Then I am planning to do an exhaustive feature selection of subsets of 3 or 4 features and input them to a Gaussian process regressor (to accomodate non-linear regressor). I will be doing feature selection and model evaluation using nested cross-validation. I hope I am taking all necessary precautions to build a model that can truly incorporate the relationship between my features and my response variable. However, because I have a huge initial feature set of 3000 features, I am afraid that I might end up picking some features that correlate well with my response variable just by luck or chance. And because I am doing exhaustive feature selection to select a smaller feature set, again chance might play a role.

What are the standard techniques to rule out the chance of luck in scenarios like this? Kindly share your inputs and links to any academic articles.

Many thanks for your time and attention!

  • $\begingroup$ Please say more about the purposes for which you "need to build a regressor." Why do you need to cut down to a subset of features, and what will you do with that subset? For example, if you're trying to develop a measure based on expression analysis of a small number of genes, issues like absolute expression levels, not just associations with outcome, might be very important. Alternatively, if you're always going to have values of all 3000 features available, keeping most or all of them might provide better predictions on future cases. $\endgroup$
    – EdM
    Jun 18, 2020 at 16:51
  • $\begingroup$ Dear EdM, I am trying to regress a clinical score from a feature set of behaviour metrics. One of my features is a correlation matrix between 2 vectors each of length 40. So, the correlation matrix itself is 800 features. Since, I have data from 60 subjects, I am skeptical that a regressor will give good results for a data set of 60 rows and 3000 columns. That's why I wanted to try feature reduction. Please correct my assumptions if they are wrong. I am new to machine learning. Your inputs will be very helpful. Thanks again $\endgroup$
    – Ed Wati
    Jun 18, 2020 at 18:09
  • $\begingroup$ Also curious what regression algorithms will work well with a dataset of 60 rows and 3000 columns. Thanks again. $\endgroup$
    – Ed Wati
    Jun 18, 2020 at 18:10

1 Answer 1


You are in a situation where the number of potential predictors (3000) far exceeds the number of cases (60), so you can't just do ordinary linear regression. With only 60 cases you probably can only include 4 to 6 effective predictors in your final model to avoid overfitting (10-15 cases per predictor). You have to find a principled way to cut down the effective number of predictors, to allow modeling without overfitting.

There are well-developed tools for dealing with such situations. An Introduction to Statistical Learning is a good resource for starting out. Much of the development of those tools was motivated by this type of situation in gene-expression studies, with thousands of potential predictors but only a few dozens of cases.

If prediction on new cases is your primary interest, it's possible to include all predictors with ridge regression. You also could consider principal-component regression (PCR) or partial least squares regression (PLS). In practice with correlated predictors those latter two approaches might keep contributions from most or all of the original predictors. In all three approaches the effective dimension of the predictor set is reduced, with coefficient values lower in magnitude than what they would be in a full model. That makes the model more likely to apply adequately to new cases.

If selection among the predictors is important, you could use LASSO to choose a small number of predictors while penalizing coefficients to avoid overfitting. There's also a hybrid of ridge and LASSO, called elastic net.

All of those approaches require choices: the number of effective predictor dimensions to keep (PCR, PLS), or a penalty term to reduce the magnitudes of regression coefficients (ridge, LASSO) and, with LASSO, the number of retained original predictors. In these approaches, you typically use cross-validation to make those choices in a way that, say, minimizes cross-validated error.

But cross-validation for feature selection without penalization, as you seem to propose, is not a good idea. If you want to get an estimate of the reliability of your modeling process, repeat the entire process on multiple bootstrapped samples of the original data set and see how well those models explain your original data set.

With LASSO you will find that the retained predictors vary among the models based on the bootstrapped samples, but that's inevitable when you have a large number of correlated predictors. Despite that variability in the identities of the retained predictors, LASSO still can work well for predicting new cases.

  • $\begingroup$ Dear EdM, Appreciate your detailed and helpful reply.I have a quick question. All the methods you suggested- LASSO, Ridge regressor are linear regressor models. Is it advisable to use non-linear regressors like Gaussian Process regression in the case of small datasets? $\endgroup$
    – Ed Wati
    Jun 24, 2020 at 8:43
  • $\begingroup$ @EdWati I don't have much experience with Gaussian process regression. You can handle nonlinearities with respect to a continuous predictor in a regression context with restricted cubic splines; linearity in linear regression is in terms of the coefficients, not the predictors themselves. The basic problem will be the same, however: you add additional predictors to your model in these ways, and if you try to fit too many parameters with a small data set your results will not generalize well unless you impose some type of penalization. $\endgroup$
    – EdM
    Jun 24, 2020 at 13:47
  • $\begingroup$ Dear EdM, Thanks again for your helpful reply!! Will try your suggestions! $\endgroup$
    – Ed Wati
    Jun 27, 2020 at 11:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.