0
$\begingroup$

I'm seeking recommendations for feature selection methods before applying a random forest model to high-dimensional data, specifically with over 60,000 features and only 1,000 samples. My concern is that directly inputting all 60,000 features into the random forest may impact prediction performance and be computationally intensive.

I've explored the Boruta algorithm, but it's too computationally expensive given the number of features. I'm also not sure about using simple filtering methods like univariate filtering, as they might overlook non-linear and complex relationships.

My current plan is to initially run a random forest model with all 60,000 features and then select the top N features (perhaps around 2,000) for further model training and tuning. I am wondering does this make sense? Btw,we've ruled out PCA due to interpretability concerns.

I'd greatly appreciate any advice!

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! What do you mean that you have ruled out PCA due to interpretability concerns? Aren’t random forest models quite difficult to interpret and, thus, rather poor model choices if that is a primary concern? Perhaps you can say more about your goals. $\endgroup$
    – Dave
    Commented Aug 19, 2023 at 3:25
  • $\begingroup$ Absolutely, Thank you! Since random forest generates the feature importance metric, we can look at which features are important. We need individual feature as we will later rely on those features to design new experiment. If we do PCA on data and run random forest, we will end up having important PCs, which is difficult to explain. $\endgroup$
    – Meow Mix
    Commented Aug 19, 2023 at 4:25

1 Answer 1

Reset to default
2
$\begingroup$

I would think twice before approaching this problem without more data. In your comment to Dave, you mention

Since random forest generates the feature importance metric, we can look at which features are important.

Karandeep Singh has a very insightful twitter thread demonstrating that feature importance scores can sometimes not yield the most important feature. For this reason, I doubt this method of feature selection (or any other for that matter, but we'll get there) will work.

It sounds like your feature selection procedure will inform some downstream decision. To me, that would mean you want to do stable feature selection (meaning that you would find the right variables each time, regardless of the sample obtained). I have a very simple simulation showing that even when all assumptions about the data generating process are met, feature selection can't reliably find the right variables. Mind you this is with stepwise regression, but I don't have much confidence in other methods.

Can you elaborate on what you intend to use the selected variables for? What is this experiment you're planning? What have you captured data on and what are your goals?

From your comment, it sounds like you'd like to:

  • Identify the genes that are associated with disease, and
  • Use those genes to predict future disease status.

I have no doubt you can create a model with the data you have. Personally, I think performing stable feature selection is a fools errand here. You have 60, 000 genes and only 1000 samples, there are many such combinations of features which could lead to reasonable prediction accuracy, even if no relationship truly exists. I think it would be a better idea to use all data and reduce the dimension of the problem using something like PCA. If there is any signal, you could be able to use the PCs to predict outcomes which would fulfill your second goal.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for sharing your insights on feature selectioe and pointing out interesting post. Our objective revolves around using omics data to predict disease phenotypes. Our primary aim is to achieve high prediction accuracy. Additionally, our secondary primary goal is to identify genes that play a crucial role in prediction. Once identified, we plan to conduct wet lab experiments to validate their association with the phenotype. $\endgroup$
    – Meow Mix
    Commented Aug 19, 2023 at 15:40
  • $\begingroup$ @MeowMix. See my edit. I think stable feature selection here is nigh impossible, esp given the fact you have 60x more variables than samples. You may be able to create a prediction model, and validating it with a new sample is a good idea, but identifying which genes are responsible (if any) is a very tough challenge with the size if the data you currently have. $\endgroup$
    – Demetri Pananos
    Commented Aug 19, 2023 at 15:59
  • $\begingroup$ +1 Readers might also be interested in Frank Harrell’s simulation of LASSO feature selection instability and his comments later in the presentation about bootstrap validation of LIME and SHAP feature importance (especially his suspicion that such validation is likely to expose those methods as lacking the desired stability, which some work of mine has suggested will happen). $\endgroup$
    – Dave
    Commented Aug 19, 2023 at 16:46
  • $\begingroup$ @DemetriPananos could you please elaborate more on "If there is any signal, you could be able to use the PCs to predict outcomes which would fulfill your second goal"? If we use Random forest with PCA, we are going to have important PCs. Do you mean we can look at the loadings of PCs to decide which genes are important? $\endgroup$
    – Meow Mix
    Commented Aug 19, 2023 at 17:57
  • 1
    $\begingroup$ More importantly, I'm saying you should eschew interpretability; even if you do identify "important" genes, their importance is unlikely to translate to a new sample. It isn't sensible to do this, so focus in prediction instead. $\endgroup$
    – Demetri Pananos
    Commented Aug 19, 2023 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.