0
$\begingroup$

I am trying to classify patients into 2 different groups using a random forest. The features correspond to the gene expression of individual patients. This means, that I have around 20.000 features (20.000 genes were measured) and 120 observations (patients).

I am wondering, if it would make sense to pre-select genes, for example based on how much they vary across patients and then use this subset, of for example 3000 genes to train the actual RandomForest. I am unsure about this, because from what I have read, RandomForest basically automatically evaluates the importance of features and drops unnecessary ones.

Another question I had was, how you would proceed if you want to predict the class of a patient using for example only the 10 most informative genes (returned by a first run of the RandomForest using all genes). Should I re-train the Classifier using only the 10 features as an input or is it possible to use my trained Classifier (trained using all genes) and only provide it with the 10 features that I am interested in (I am using scikit-learn for the implementation).

Any help or comments are much appreciated!

Cheers!

$\endgroup$
3
  • $\begingroup$ A principled way to use random forest for feature selection is boruta. $\endgroup$
    – Sycorax
    Apr 8, 2022 at 14:02
  • 1
    $\begingroup$ Why do feature selection at all when at each split (in each tree) the random forest algorithm itself will select the "best" feature to split on? $\endgroup$
    – dipetkov
    Apr 9, 2022 at 18:46
  • $\begingroup$ I thought that since I have a lot of uninformative features (genes) that some trees will never see any meaningful features or only a few. So I wanted to increase the likelihood that these genes are selected $\endgroup$
    – nhaus
    Apr 10, 2022 at 18:20

1 Answer 1

0
$\begingroup$

I believe refitting a trained random forest with sufficient number of deep trees on a subset of high-impurity features will reduce variance and introduce minor bias. But is it worth computationally - training two ensembles on thousands of features? I'd rather combine tree-based ensembles with, say, linear models like RidgeClassifier or LogisticRegression(penalty='l1'), having selected $p << n$ or $p \leq n$ features, respectively (although I don't have theoretical justifications of combining impurity- and residual-based models).

is it possible to use my trained Classifier (trained using all genes) and only provide it with the 10 features that I am interested in.

You should provide exactly the same number of features every time you make predictions, otherwise, scikit-learn will throw an exception. If you want your final predictor to use only informative features, consider using SelectFromModel along with Pipeline.


Speaking of linear models with $L_1$ penalty and similar models updating coefficients coordinate-wise by applying soft-thresholding, the nonzero coefficients tend to be biased toward zero, so debiasing the model by refitting LinearRegression or Ridge with smaller penalty on the chosen feature subset makes sense:

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import Lasso, Ridge

model = make_pipeline(SelectFromModel(Lasso()), Ridge(alpha=1e-4))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.