2
$\begingroup$
  1. I have 309 samples [tumors] and 3234 features [genes]. I used scikit-learn python library to run random forest with one parameter n_estimators=100.
  2. I also used train_test_split to spit my dataset into 70-30.
  3. When I run the model several times, i.e, each time - randomly split data as 70-30 and predict feature importances; I get different features ranked as important. Sometimes there is an overlap while most of the times there is none - often new features that were NOT found in the original run show up.
  4. Also the most top 10 ranked feature scores are within 0.01-0.03.
  5. There are highly correlated features in this dataset (as genes are often co-regulated - or, features are often inter-linked in an organic network).

Is this commonly noted and if so, are there ways to come to a consensus important feature by averaging the "feature_importance_score" across 10-20 random runs?

If this is not common, any suggestions where I may be going wrong?

$\endgroup$
  • 1
    $\begingroup$ One hundred trees is way too few. Try 1000, or even 10k. You're dealing with large number of combinatoric possibilities. $\endgroup$ – horaceT Sep 30 '17 at 17:39
  • $\begingroup$ ^ This. You need many, many more trees. $\endgroup$ – Matthew Drury Nov 3 '17 at 17:25
1
$\begingroup$

I'm right now using R to do something similar.

So if there is a huge change in feature importance in RandomForest this is due to random-number-generation influence. Maybe the 70-30 split is not always the same. And depending on the RandomForest implementation it will use randomness for training.

Your training data sounds like it could overfit, if you have just 309 observations with 3234 features each. Maybe you can get more data?

I think your approach is fine. It can work. But maybe the result is just, that all features are pretty much the same important on the given training set - and then you need a different approach.

$\endgroup$
  • $\begingroup$ This is data published by The cancer genome atlas (TCGA). It is quite expensive to add more samples and likely the number may not increase in the near future. In regards to the overfit - I'm exploring feature reduction such as RFE (recursive feature elimination). I will post those results soon $\endgroup$ – Tejaswi Iyyanki Oct 28 '16 at 18:46
1
$\begingroup$

In a random forest, the building of individual trees is actually based on randomness. So in each run, different variables get chosen in each tree and decision point.

If I'm not mistaken, setting your seed before running random forest could lead to repeatable outputs.

And as @Tobi pointed out, the ratio of observations to features in your dataset is cause for concern with respect to overfitting.

$\endgroup$
  • $\begingroup$ I always thought that the random forest results were stable overall. In other words - I would get similar rank of feature importances even if the underlying feature_importance_values may vary with each random run of the same data (assuming I have enough "number of samples"). If it indeed changes - do you think I can run random forest on 100 random splits of my data @ (70-30) and then get an average feature_importance_scores, and use that to feel confident of the ranking? $\endgroup$ – Tejaswi Iyyanki Oct 28 '16 at 18:52
  • $\begingroup$ As already mentioned, the reason your importance varies could be lack of data $\endgroup$ – Arun Jose Oct 29 '16 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.