3
$\begingroup$
  1. I have 309 samples [tumors] and 3234 features [genes]. I used scikit-learn python library to run random forest with one parameter n_estimators=100.
  2. I also used train_test_split to spit my dataset into 70-30.
  3. When I run the model several times, i.e, each time - randomly split data as 70-30 and predict feature importances; I get different features ranked as important. Sometimes there is an overlap while most of the times there is none - often new features that were NOT found in the original run show up.
  4. Also the most top 10 ranked feature scores are within 0.01-0.03.
  5. There are highly correlated features in this dataset (as genes are often co-regulated - or, features are often inter-linked in an organic network).

Is this commonly noted and if so, are there ways to come to a consensus important feature by averaging the "feature_importance_score" across 10-20 random runs?

If this is not common, any suggestions where I may be going wrong?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ One hundred trees is way too few. Try 1000, or even 10k. You're dealing with large number of combinatoric possibilities. $\endgroup$
    – horaceT
    Sep 30, 2017 at 17:39
  • $\begingroup$ ^ This. You need many, many more trees. $\endgroup$
    – Matthew Drury
    Nov 3, 2017 at 17:25

2 Answers 2

Reset to default
3
$\begingroup$

I'm right now using R to do something similar.

So if there is a huge change in feature importance in RandomForest this is due to random-number-generation influence. Maybe the 70-30 split is not always the same. And depending on the RandomForest implementation it will use randomness for training.

Your training data sounds like it could overfit, if you have just 309 observations with 3234 features each. Maybe you can get more data?

I think your approach is fine. It can work. But maybe the result is just, that all features are pretty much the same important on the given training set - and then you need a different approach.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is data published by The cancer genome atlas (TCGA). It is quite expensive to add more samples and likely the number may not increase in the near future. In regards to the overfit - I'm exploring feature reduction such as RFE (recursive feature elimination). I will post those results soon $\endgroup$
    – Tejaswi Iyyanki
    Oct 28, 2016 at 18:46
3
$\begingroup$

In a random forest, the building of individual trees is actually based on randomness. So in each run, different variables get chosen in each tree and decision point.

If I'm not mistaken, setting your seed before running random forest could lead to repeatable outputs.

And as @Tobi pointed out, the ratio of observations to features in your dataset is cause for concern with respect to overfitting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I always thought that the random forest results were stable overall. In other words - I would get similar rank of feature importances even if the underlying feature_importance_values may vary with each random run of the same data (assuming I have enough "number of samples"). If it indeed changes - do you think I can run random forest on 100 random splits of my data @ (70-30) and then get an average feature_importance_scores, and use that to feel confident of the ranking? $\endgroup$
    – Tejaswi Iyyanki
    Oct 28, 2016 at 18:52
  • $\begingroup$ As already mentioned, the reason your importance varies could be lack of data $\endgroup$
    – Arun Jose
    Oct 29, 2016 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.