I have a theoretical question..

  1. I have a model, let it be random forest

  2. I take 100 candidate features and train the model

  3. I select all the features that are important from the point of view of the model (important features from the random forest). Let the model select 40 features. I'll call it the basic feature set

My question is: Can it be that the model will work better if I choose some subset of the basic features set of in 5-10-20 features?

Is it worth looking for a subset of features in the base feature set, or is it already the best set by default?

  • 1
    $\begingroup$ You would do well to place no confidence in the 40 features. Bootstrap confidence intervals of the 100 features will uncover the true difficulty of the feature selection task. These confidence intervals will be embarrassingly wide. $\endgroup$ Commented Jul 9, 2023 at 15:49
  • 1
    $\begingroup$ I don't really understand how to calculate and analyze these confidence intervals. If you can provide an example code in R , that would be awesome. $\endgroup$
    – mr.T
    Commented Jul 9, 2023 at 16:26
  • 1
    $\begingroup$ An R example is here and see also this. $\endgroup$ Commented Jul 10, 2023 at 12:26

1 Answer 1


In theory, no. Because the mutual information between the target and the features decreases or stays the same when you take out features. The following rule in information theory describes this:

$$I(Y; X_1, X_2) = I(Y; X_1) + I(Y; X_2|X_1)\longrightarrow I(Y;X_1,X_2)\geq I(Y; X_1)$$

But, in practice, yes. No model or algorithm is perfect and they may work better with less features. Part of the reason is the algorithm or the implementation itself, and the other part is the limited capability of us doing hyper-parameter tuning. For example, in the case of random forests, note that the best hyper parameters may change with respect to number of features, e.g. the tree depth. The search space is combinatorially large and usually it is difficult to exhaust. Finally, there is the effect of stochasticity in the process.

Another point is that you are selecting the features that the algorithm thinks important. So, it's a feedback cycle and may not work well always. The same could have been done in a simple algorithm like logistic regression and could result in poor performance.

Is it worth looking for a subset of features in the base feature set, or is it already the best set by default?

The short answer is Yes, but only if you suspect that your algorithm is not working well with that set of features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.