Suppose I have trained a random forest on a very high dimensional set of features. Then I have identified the top m features that have the highest Gini index. If I'd now drop all other feature variable except the top m ones (so performing a great dimensionality reduction), can I expect to obtain a similar classification?


1 Answer 1


I don't think this question can be meaningfully answered, since this -- obviously -- depends on your data set. For example, you might have a single variable which is a perfect classifier; or you might need at least a thousand features to meaningfully classify your data. What do I know?

However, two things can be noted.

  1. You can (and should) test how feature selection influences your models performance. Mind that to do it meaningfully, you should use a LOO-CV or KFOLD-CV testing scheme, as you will not be able to rely on the OOB (out of the bag) error estimates.

  2. The maximum meaningful number of features will depend on the number of samples. The number of features should most likely be much smaller than the number of samples; increasing number of features beyond the number of samples will not be likely to improve the performance.

  • $\begingroup$ My question was more along the lines: Suppose I have 10 features which Gini index identifies as very important and then there is a stark jump and all the other features have a Gini score indicating that they are not really relevant. If I know leave these unimportant features out and retrain my random forest on the 10 important features, how much would I expect my decision boundaries to vary compared to the original random forest that was trained on all features? I expect only a small variation as well? $\endgroup$
    – user52145
    Apr 26, 2017 at 11:05
  • $\begingroup$ I would have supposed that at least qualitative statements like these can be made without the dataset. $\endgroup$
    – user52145
    Apr 26, 2017 at 11:06
  • 1
    $\begingroup$ I think my answer, independent of the jumps in the Gini index, remains: you need to test this. However, you might want to read this answer: stats.stackexchange.com/questions/92419/… $\endgroup$
    – January
    Apr 26, 2017 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.